Metadata-Version: 2.5
Name: py-yonder
Version: 0.2.8
Summary: Run Python functions inside containers via a decorator.
Project-URL: Homepage, https://jason-berger.gitlab.io/yonder/
Project-URL: Documentation, https://jason-berger.gitlab.io/yonder
Project-URL: Repository, https://gitlab.com/jason-berger/yonder
Project-URL: Issues, https://gitlab.com/jason-berger/yonder/-/issues
Project-URL: Source, https://gitlab.com/jason-berger/yonder
Author-email: Jason Berger <berge472@gmail.com>
License: MIT
Keywords: cloudpickle,container,decorator,docker,remote-execution
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.9
Requires-Dist: cloudpickle>=3.0
Provides-Extra: all
Requires-Dist: docker>=7.0; extra == 'all'
Requires-Dist: fastapi>=0.110; extra == 'all'
Requires-Dist: kubernetes>=28.0; extra == 'all'
Requires-Dist: requests>=2.28; extra == 'all'
Requires-Dist: uvicorn>=0.29; extra == 'all'
Provides-Extra: api
Requires-Dist: requests>=2.28; extra == 'api'
Provides-Extra: dev
Requires-Dist: docker>=7.0; extra == 'dev'
Requires-Dist: fastapi>=0.110; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: kubernetes>=28.0; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: requests>=2.28; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: twine>=7; extra == 'dev'
Requires-Dist: uvicorn>=0.29; extra == 'dev'
Provides-Extra: docker
Requires-Dist: docker>=7.0; extra == 'docker'
Provides-Extra: k8s
Requires-Dist: kubernetes>=28.0; extra == 'k8s'
Provides-Extra: server
Requires-Dist: fastapi>=0.110; extra == 'server'
Requires-Dist: uvicorn>=0.29; extra == 'server'
Description-Content-Type: text/x-rst

.. image:: doc/assets/images/logo.svg
   :alt: yonder
   :align: center
   :width: 380px

|

yonder
======

Run Python functions on container-backed runners via a decorator. Runners only need ``sh`` and ``python`` on ``PATH``; yonder handles the rest.


Install
-------

.. code:: bash

    # core package
    pip install py-yonder

    # with all extras (DockerRunner, K8sRunner, ApiRunner, etc.)
    pip install py-yonder[all]

Quick start
-----------

.. code:: python

    import yonder
    from yonder import DockerRunner, missing

    # Existing image#
    runnerA = DockerRunner(
        image="url/to/image",
        env={"ENV_VARA": "value"},
        workspaces={"/host/path": "/in/container"},
        when=missing("liba"),  # dispatch only when host lacks `liba`, otherswise run locally
    )

    # Build from a Dockerfile (context defaults to the Dockerfile's directory).
    runnerB = DockerRunner(
        dockerfile="./docker/myimg.Dockerfile",
        build_args={"VERSION": "1.2.3"},
    )

    # On-demand: fresh container per call instead of one long-lived container.
    ephemeral = DockerRunner(image="url/to/image", on_demand=True)

    # Attach to a container you manage outside yonder. yonder will start it
    # if it's stopped, but never removes it on close.
    external = DockerRunner(container="my-dev-container")

    # Register them by name.
    yonder.register({
        "runnerA": runnerA,
        "runnerB": runnerB,
        "ephemeral": ephemeral,
        "external": external,
    })

    # Decorate. Dispatch policy lives on the runner (via the runner's
    # `when=`); the decorator just names the target.
    @yonder.to("runnerA")
    def functionA(a, b):
        return a + b

    @yonder.to("runnerB")
    def functionB(a, b):
        return a * b

Lazy start by default
~~~~~~~~~~~~~~~~~~~~~

A registered runner is **not** built/started until it's actually needed.
The first decorated call that targets a runner triggers its ``start()``
(image build/pull, persistent container creation). Runners that are never
called incur no cost.

You can pre-warm explicitly when you'd rather pay startup cost up front:

.. code:: python

    runnerA.start()       # pre-warm a specific runner
    yonder.wait()         # pre-warm every registered runner (parallel)

Both are opt-in. On-demand runners always behave the same way — they spin
up a fresh container per call regardless of pre-warming.

Tear down
~~~~~~~~~

Containers/Pods created by yonder are cleaned up automatically on process exit, but you can also tear down explicitly:

.. code:: python

    runnerA.close()       # tear down a specific runner
    yonder.closeAll()     # tear down every registered runner (parallel)


DockerRunner options
--------------------

.. code:: python

    DockerRunner(
        # Exactly one of these three is required:
        image=None,          # use/pull an existing image; yonder creates the container
        dockerfile=None,     # build an image locally first; yonder creates the container
        container=None,      # attach to an existing container (managed outside yonder)

        build_context=None,  # override the build context directory (dockerfile mode)
        build_args=None,     # dict of Docker ARGs (dockerfile mode)

        env=None,            # environment variables (image/dockerfile mode only)
        ports=None,          # publish container ports to the host, like
                             # `docker run -p`. Orientation is container:host.
                             #   8080                       # host 8080 -> container 8080
                             #   "8080:3333"                # host 3333 -> container 8080
                             #   [8080, "9000:9001/udp"]    # list; per-port protocol
                             #   {8080: 3333}               # dict, {container: host}
                             #   {8080: None}               # ephemeral host port
                             #   {8080: ("127.0.0.1", 3333)}# bind a host IP
                             # image/dockerfile mode only (rejected with container=).
                             # Read assigned ports back via .published_ports.
        workspaces=None,      # declarative file sync. Accepts a list, a dict,
                             # or a mix:
                             #   ["host/path/a", "host/path/b"]       # list-form, auto-mapped
                             #   {host: runner_path}                  # dict, explicit runner path
                             #   {host: {to?, dir?, include?,         # dict, full opts ("to" optional;
                             #           exclude?, name?}}            #   omit it to auto-map)
                             #   [host, {host: {dir: "up"}}, ...]     # list of strings + dicts
                             # In image=/dockerfile= mode, simple entries
                             # auto-promote to bind mounts.
        sync_deletes=False,  # propagate deletions across sync (default: False)
        on_demand=False,     # one-shot container per call (image/dockerfile mode only)
        inject_cloudpickle=False,  # derive `FROM image + RUN pip install cloudpickle`
                                   # (image mode only; tagged by base-image hash so the
                                   # docker layer cache makes re-runs cheap)
        mode="reference",    # transport: "cloudpickle" or "reference"
        source_path=None,    # host path(s) made importable in the container.
                             # In mode="reference", None defaults to the caller's
                             # sys.path root (or just the caller's package, if it
                             # is installed in site-packages); pass [] to opt out
        source_transport="auto",
                             # how source_path reaches the container:
                             #   "mount" — bind mount at /workspaces/<name>
                             #   "ship"  — tar + upload into the container
                             #   "auto"  — ship when yonder itself runs inside a
                             #             container, else mount
                             # container= always ships (yonder can't add mounts
                             # to a container it didn't create)
        source_include=(".py", ".yaml", ".yml", ".json"),
                             # extension allowlist for shipped tar (applies
                             # whenever sources are shipped); None = every file
        client=None,         # pre-built docker.DockerClient (else docker.from_env())
    )

K8sRunner
---------

Run inside a Kubernetes pod. Three modes — yonder either creates the
pod from an image, or attaches to one you already manage.

.. code:: python

    from yonder import K8sRunner

    # Create a pod from an image. Yonder owns it: close() deletes it.
    owned = K8sRunner(
        image="myorg/worker:1.2.3",
        namespace="default",
        env={"LOG_LEVEL": "INFO"},
        image_pull_secrets=["my-registry-secret"],  # private registries
    )

    # Attach by pod name.
    by_name = K8sRunner(
        pod="my-pod-abc123",
        namespace="default",
        container="app",  # optional; defaults to the pod's first container
    )

    # Attach via label selector — first Running pod that matches is used.
    by_selector = K8sRunner(
        selector="app=my-app,env=dev",
        namespace="default",
    )

For anything beyond what ``image=`` exposes (resources, service account,
tolerations, custom volumes, sidecars, …) deploy the pod with your own
tooling and attach via ``pod=`` / ``selector=``.

.. code:: python

    K8sRunner(
        # Exactly one of these three is required:
        pod=None,            # attach by pod name (yonder doesn't manage lifecycle)
        selector=None,       # attach by label selector (yonder doesn't manage lifecycle)
        image=None,          # create a pod from this image (yonder owns its lifecycle)

        namespace="default", # pod namespace
        container=None,      # container within the pod (default: first)
        kubeconfig=None,     # kubeconfig path (else KUBECONFIG/~/.kube/config,
                             # falling back to in-cluster config)
        context=None,        # kubeconfig context
        name=None,           # runner name (auto-derived if omitted)
        mode="cloudpickle",  # transport: "cloudpickle" or "reference"
        source_path=None,    # host path(s) tarred and shipped into the pod
                             # under /tmp/yonder-src/<sha>/ (cached by digest).
                             # In mode="reference", None defaults to the
                             # caller's __file__ dir; pass [] to opt out
        source_include=(".py", ".yaml", ".yml", ".json"),
                             # extension allowlist for shipped tar;
                             # None = ship every file
        ports=None,          # reach pod ports on localhost via port-forward
                             # (the kubectl port-forward equivalent). Same forms
                             # as DockerRunner; orientation is pod_port:local_port.
                             # tcp only. Works in every mode (image/pod/selector).
                             #   {8080: None}  # OS-chosen local port
                             # Read assigned ports back via .forwarded_ports.

        # image= only:
        env=None,                  # dict of env vars to set on the container
        image_pull_secrets=None,   # list of existing Secret names
        startup_timeout=60,        # seconds to wait for pod to reach Running

        api_client=None,     # pre-built kubernetes.client.ApiClient
    )

The target container must have ``python`` on ``PATH``, plus whatever
the runner's transport mode needs — see `Container requirements`_ below.

Architecture
------------

- **Runner** (``yonder.Runner``) — abstract base. Implementations hold
  their own config and expose ``run(payload) -> bytes``, plus optional
  ``start()`` / ``close()`` lifecycle hooks.
- **DockerRunner** — default implementation; uses the
  `Docker SDK for Python <https://docker-py.readthedocs.io/>`_. Supports
  existing images, Dockerfile builds, and attaching to externally-managed
  containers; persistent and on-demand modes.
- **K8sRunner** — Kubernetes implementation; uses the
  `Kubernetes Python client <https://github.com/kubernetes-client/python>`_
  to exec into a pod selected by name or label selector.
- **YonderSession** — process-wide name→runner registry. Populate with
  ``yonder.register({...})``, bring everything online with
  ``yonder.wait()``, tear everything down with ``yonder.closeAll()``.
- **Decorator** — serializes ``(func, args, kwargs)`` with cloudpickle,
  calls the named runner's ``run(payload)``, deserializes the envelope, and
  either returns the value or re-raises the exception.

Writing your own Runner
-----------------------

.. code:: python

    from yonder import Runner, register

    class MyRunner(Runner):
        def __init__(self, **config):
            ...

        def start(self):
            # eager init (build/pull image, warm container, etc.)
            ...

        def run(self, payload: bytes) -> bytes:
            # send payload to your execution environment,
            # return the cloudpickled envelope
            ...

        def close(self):
            ...

    register({"mine": MyRunner(...)})

Container requirements
----------------------

Every runner shells out to ``python`` inside the target environment, so
the image, container, or pod must have ``python`` on ``PATH``. Beyond
that, the requirements depend on which **transport mode** the runner
uses to ship your function across the wire — not on which runner you're
using. Both ``DockerRunner`` and ``K8sRunner`` accept
``mode="cloudpickle"`` (default) or ``mode="reference"`` and have
identical environment requirements for each.

Cloudpickle mode
~~~~~~~~~~~~~~~~

``mode="cloudpickle"`` (the default) serializes the function as bytecode
with cloudpickle and unpickles it on the other side. Works for closures,
lambdas, locally-defined functions, and code typed at a REPL — nothing
needs to be importable in the container.

The target environment must have:

- ``cloudpickle`` installed (``pip install cloudpickle``).
- The same Python **minor** version as the host (e.g. host 3.11 ↔
  container 3.11). CPython bytecode is not portable across minor
  versions; yonder raises a clear error on mismatch.

If a base image doesn't ship with cloudpickle, ``DockerRunner`` can
inject it for you — pass ``inject_cloudpickle=True`` and yonder will
build a tiny derived image (``FROM <your-image>`` +
``RUN pip install cloudpickle``) and cache it locally. Injection isn't
available when attaching to an existing container (``container=``) or
when building from a Dockerfile — add ``RUN pip install cloudpickle`` to
the Dockerfile yourself in that case. ``K8sRunner`` never modifies the
pod, so cloudpickle must already be present in the target container.

Reference mode
~~~~~~~~~~~~~~

``mode="reference"`` only ships ``(module, qualname, args, kwargs)``.
The container imports its own copy of the function and calls it —
bytecode never crosses the wire, so host and container Python versions
may differ, and cloudpickle is not required in the container.

The target environment must have:

- The function's module importable inside the container. The simplest
  way is to pass ``source_path=`` to the runner — yonder will get the
  source tree there for you. On ``DockerRunner`` with ``image=`` /
  ``dockerfile=``, that's a bind mount at ``/workspaces/<basename>``.
  On ``DockerRunner`` with ``container=`` or ``K8sRunner``, the tree
  is tarred and shipped to ``/tmp/yonder-src/<sha>/`` on the first
  call (cached by content digest, so an unchanged tree only ships
  once per runner). Either mechanism also helps cloudpickle mode if
  the function does ``import helpers`` against a sibling.

  See `Running yonder inside a container`_ when the *calling* process
  is itself containerized — bind mounts don't work from there.

Reference mode requires the function to be defined at module scope in a
non-``__main__`` module; closures, lambdas, and REPL-defined functions
won't work.

Shipping requires ``sh`` and ``tar`` on the target's ``PATH``;
distroless / minimal images that strip them are incompatible. Pre-bake
the source or use a different base image. (``DockerRunner`` is the
exception — it uploads through the Docker API and needs neither.)

.. _Running yonder inside a container:

Running yonder inside a container
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When your application is itself containerized and talks to the host's
Docker socket, **bind mounts don't work**. Docker resolves a mount
source against the *daemon's* filesystem, not your container's, so a
path like ``/app/mypkg`` — real inside your container, absent on the
host — is silently created as an empty directory and mounted. Reference
mode then fails with ``ModuleNotFoundError: No module named 'mypkg'``.

``DockerRunner`` detects this and handles it automatically. When the
client is containerized it inspects *its own* container's mounts and
works out, per path, whether the daemon can actually see it:

- **Under one of your container's bind mounts** — the path is rewritten
  to that mount's host-side source and mounted as usual. You keep the
  zero-copy, live-shared bind, and the two sides no longer have to be
  the *same* path, so the ``-v /srv/src:/srv/src`` same-path trick is no
  longer needed.
- **Anywhere else** (inside your image, as an installed package is) —
  the contents are tarred and uploaded through the Docker API instead.

``source_transport="auto"`` (the default) applies that to ``source_path``;
force it either way with ``source_transport="ship"`` / ``"mount"``. The
important part is that yonder never silently mounts a path the daemon
cannot resolve, which is what produced the empty directory.

The second half of the problem is *what* gets shipped. In a container
your application is usually ``pip install``-ed, so the ``sys.path`` root
``SourcePath.AUTO`` resolves to is ``site-packages``. Shipping that
whole tree would put every host-installed package ahead of the runner
image's own copies on ``sys.path`` — and since ``source_include``
defaults to source files only, compiled dependencies would lose their
``.so`` files and stop importing.

So when AUTO lands in ``site-packages`` (or ``dist-packages``), yonder
ships only the caller's top-level package:

.. code:: python

    # /usr/local/lib/python3.12/site-packages/mypkg/tasks.py
    import yonder as yn

    @yn.to("box")
    def crunch(n):
        return n * 2

``mypkg/`` alone is uploaded, landing at
``/tmp/yonder-src/<sha>/pkgroot/mypkg/`` with ``.../pkgroot`` on the
runner's ``sys.path`` — so ``import mypkg.tasks`` resolves and nothing
else in your site-packages shadows the runner's environment. Your
application's third-party dependencies still have to be present in the
runner image, exactly as before.

Pass a ``PackageDir`` to get the same treatment for any package
directory explicitly:

.. code:: python

    from yonder import DockerRunner, PackageDir

    DockerRunner(image="python:3.12-slim",
                 source_path=PackageDir("/usr/local/lib/python3.12/site-packages/mypkg"))

A plain string ``source_path`` keeps its original meaning: the
directory itself goes on the runner's ``sys.path``.

.. warning::

   ``SourcePath.AUTO`` resolves from the file that **constructs the
   runner**, not the one that defines the function. If your app builds
   its ``DockerRunner`` in an entrypoint script that sits outside the
   package (``/app/main.py``, a ``console_scripts`` wrapper), AUTO
   resolves to *that* directory and your package still won't be
   importable in the runner. Construct the runner from inside the
   package, or pass the package explicitly:
   ``source_path=PackageDir(os.path.dirname(mypkg.__file__))``.

``workspaces=`` entries are resolved the same way. Normally yonder
promotes a simple workspaces entry to a bind mount because that is
strictly faster; when the client is containerized it does so only for
entries the daemon can see (translating the source), and leaves the rest
on the sync path, where the files are read from *your* filesystem and
uploaded. The trade-off is semantic rather than functional: a bind is
live-shared, while a synced workspace is snapshotted in before the call
and back out afterwards.

.. note::

   Live bidirectional sharing is impossible for a path the daemon cannot
   reach — there is no filesystem to share. If a workspace must be
   live, make it reachable by mounting it into your own container (any
   destination will do; yonder translates), rather than relying on a
   path that only exists in your image.
