Metadata-Version: 2.4
Name: sontris-raytest
Version: 0.0.3
Summary: Ray-backed remote execution and reusable pytest support
License-Expression: LGPL-3.0-or-later
License-File: COPYING
License-File: LICENCE.md
Requires-Python: >=3.10
Requires-Dist: pytest>=9.2.0
Requires-Dist: ray[client,default]
Requires-Dist: typing-extensions>=4.0
Provides-Extra: testing
Requires-Dist: pytest>=9.2.0; extra == 'testing'
Description-Content-Type: text/markdown

# sontris-raytest: Ray-backed remote execution and pytest support

`sontris-raytest` is a Python library for executing scripts and commands on Ray
workers. It also ships reusable pytest fixtures, test helpers, and a sample
remote project. The user and API documentation is in [`docs/`](docs/index.rst);
build the HTML site with Sphinx using the instructions in
[`docs/contributing.rst`](docs/contributing.rst).

1. The local project installs sontris-raytest and its Ray Client dependencies.
2. Pytest connects to the configured Ray head and creates a worker-side
   workspace from the selected project directory.
3. By default, the worker runs `uv sync` for that project, creating a separate
   remote virtual environment.
4. Tests execute commands and scripts using the selected remote environment
   and download generated artifacts.
5. A Rust integration test provisions a toolchain if needed, compiles a
   standard-library-only HTTP calculator on the worker, and runs an API
   testharness there.

The local pytest environment and the remote sample project's environment are
intentionally separate. The tests verify that remote setup succeeds and that
the sample project's `python-dotenv` dependency is importable from its virtual
environment. The Rust example is separate from that Python environment: its
source is uploaded with the fixture, then built and tested on the worker.

## Requirements and setup

- Python 3.10+
- [uv](https://docs.astral.sh/uv/) on workers when using the default remote
  environment setup
- A Ray cluster reachable over Ray Client

The local Ray Client Python version must match the Python version used to
start the cluster. This checkout pins uv to Python 3.13.5 in
`.python-version`, matching the example remote head. If connecting to a head
started with another Python version, update the pin with `uv python pin
<version>` and recreate the local environment with `uv sync --locked`.
`requires-python = ">=3.10"` remains the package compatibility range; the
pin selects the interpreter for this development environment.

Install the development environment:

```bash
uv sync --locked
```

External test suites can install the test support with:

```bash
pip install "sontris-raytest[testing]"
```

In their `conftest.py`, load the packaged pytest fixtures:

```python
pytest_plugins = ["sontris.testing.raytest.pytest_plugin"]
```

The plugin uploads the nearest project root from the current directory by
default. Select a different project directory with
`pytest --sontris-raytest-working-dir=/path/to/project`, or override its
`sontris_raytest_working_dir` fixture. The sample project exposed as
`sontris.testing.raytest.REMOTE_PROJECT` is useful for examples and
sontris-raytest's own tests; it is not forced on consuming projects.

Then import the common helpers in tests with
`from sontris.testing.raytest import execute_script, run_command`.

## Versioning and wheel traceability

The package version is dynamic: Hatchling obtains it from Git through the
build dependency `uv-dynamic-versioning==0.14.1`, as configured in
`pyproject.toml`. The tool uses the latest matching PEP 440 Git tag;
unprefixed tags such as `1.2.3` and `v`-prefixed tags such as `v1.2.3` are
accepted. A build at a release tag gets that tag's version. Commits after a
tag use a bumped PEP 440 development version, including Git revision
information.

The configuration also defines `fallback-version = "0.0.0"`. This is a
baseline for builds where no matching tag is available; with the configured
version bumping and available commit metadata, the resulting wheel can be a
development version rather than literally `0.0.0`. Therefore, do not infer
the version from the latest release tag alone: inspect the wheel produced by
the build. Build from a Git checkout with the relevant tags and history
available. CI checks out full history (`fetch-depth: 0`). After checks pass, a
push of a non-development release tag builds, verifies, and publishes the wheel
and source distribution to PyPI. The workflow rejects development and local
versions and verifies that the wheel version matches the tag. Branch pushes
do not publish to PyPI. After a release tag passes checks and is published to
PyPI, the workflow triggers the Read the Docs ``latest`` build and creates a
Forgejo release with non-merge commits since the previous tag, a
version-specific PyPI link, and a link to the English ``latest`` documentation.
A shallow clone may not have enough history to calculate the development
version correctly; fetch the full history as well as tags before building.

After committing a clean working tree, `uv run run.py tag` builds a temporary
wheel using this versioning configuration and creates a lightweight local tag
from its release version (the development and local suffixes are removed).
Push the tag to trigger the release workflow; the command does not push it.

For an auditable build, retain the following alongside the wheel:

* repository URL, full `git rev-parse HEAD` commit, tag (if any), and whether
  `git status --porcelain` was empty;
* wheel filename and the `Version` field in its `.dist-info/METADATA`;
* SHA-256 checksum of the wheel (and source distribution, if retained);
* hashes of `pyproject.toml` and `uv.lock`, plus the `uv` and Python versions
  used to build it.

Build the artifacts with the repository task:

```bash
uv run run.py build
git rev-parse HEAD
git describe --tags --long --always --dirty
git status --porcelain
git tag --points-at HEAD
uv --version
python --version
sha256sum pyproject.toml uv.lock dist/sontris_raytest-*
```

The wheel filename and its embedded `Version` field are the authoritative
package version. A development version may include a short Git revision, but
record the full commit separately; the version string alone is not a complete
source or build record. Record the build output and checksums with the
artifacts, and verify that the working tree is clean for release builds.

The local cluster defaults to `127.0.0.1` and Ray Client port `10001`. To
override them for a single command, set process environment variables:

```bash
RAY_HOST=127.0.0.1 RAY_CLIENT_PORT=10001 ./scripts/cluster/start-head.sh
```

For client processes, `RAY_HOST` is the reachable Ray Client host and
`RAY_CLIENT_PORT` is the port the client connects to. In
`scripts/cluster/start-head.sh`, `RAY_HOST` instead selects the node IP Ray
advertises. For exactly `127.0.0.1` or `localhost`, the script lets Ray choose
the local interface; for other values it passes the configured IP as
`--node-ip-address`. The script accepts an explicit IP argument or
`--ssh-address` to infer the server-side IP from `SSH_CONNECTION`. Its
separate `RAY_GCS_PORT` (default `6379`) configures Ray's GCS port, not the Ray
Client port. The dashboard is bound to `127.0.0.1:8265`, independently of the
advertised node address.

The scripts and executor do not load `.env` files. The executor reads
`RAY_HOST` and `RAY_CLIENT_PORT` from the process environment; direct
`RemoteExecutor` construction can instead use its `host` and `port` arguments.
`.env` files remain excluded from uploaded archives. Never put credentials in
the sample project or files uploaded to the cluster.
The executor does not take ownership of an already initialized Ray session by
default. For the pytest fixtures, use
`--sontris-raytest-use-existing-ray-session` or override the
`sontris_raytest_manage_ray_lifecycle` fixture to `False`. For direct use, pass
`manage_ray_lifecycle=False`. The caller must ensure the existing session is
connected to the intended cluster and is responsible for shutting it down.

## Run tests

All project tests run through the nested task group. Run Ray-free unit tests
with:

```bash
uv run run.py test unit
```

The unit task also reports line and branch coverage for the library, task
runner, and docstring checker, including source lines not exercised by tests.

Start a local Ray head and run the localhost integration suite with:

```bash
uv run run.py deploy local
uv run run.py test local
uv run run.py deploy stop-local
```

For configured remote Ray clusters, pass the reachable host for each named
cluster. Use the same `--ray-host NAME=HOST` option for remote tests and
remote deployment; deployment hosts must also be reachable over SSH. The
runner adds configured hosts to both `NO_PROXY` and `no_proxy` for subprocesses
while preserving existing entries:

```bash
uv run run.py deploy remote \
  --ray-host plt01=192.0.2.10 \
  --ray-host plt02=192.0.2.11
uv run run.py test remote \
  --ray-host plt01=192.0.2.10 \
  --ray-host plt02=192.0.2.11
uv run run.py test remote-docker \
  --ray-host plt01=192.0.2.10 \
  --ray-host plt02=192.0.2.11
```

`test remote-docker` requires local Docker access, Ray Client connectivity to
both targets, and Docker installed and usable by the selected Ray workers.
It does not install Docker remotely. The example addresses above are
documentation placeholders; provide the addresses reachable in your network.
Use `uv run run.py test collect` to collect tests without connecting to Ray.

Run every suite with managed local and remote Ray nodes using:

```bash
uv run run.py test all \
  --ray-host plt01=ray-a.example \
  --ray-host plt02=ray-b.example \
  --ssh-password-prompt
```

This command reports endpoint health, restarts the local head and both remote
heads, runs unit, local, remote, and Docker tests, then attempts to stop every
node it started even if a test fails. It requires SSH access to both hosts and
Docker usable locally and by remote workers. Starting a head stops existing
Ray processes, so run this only when restarting and stopping these nodes is
intended.

## Deploy Ray heads

Pass remote SSH targets as repeated `--ray-host NAME=HOST` options. Each host
must be reachable both as a Ray Client endpoint and over SSH; SSH aliases from
`~/.ssh/config` are supported. The remote start script discovers the host-side
IP used by the SSH connection and passes it to Ray as the advertised node
address. Configure SSH access, then deploy the cluster scripts and restart Ray
on each host using:

```bash
uv run run.py deploy remote \
  --ray-host plt01=ray-a.example \
  --ray-host plt02=ray-b.example
```

The deployment installs `uv` for the SSH user when needed and uploads
`scripts/cluster/`, including its Ray-only `pyproject.toml` and lock file, as
well as the repository's `.python-version` pin. The lifecycle scripts use that
pin for Ray head and client compatibility. `uv run run.py deploy local`
starts the local head at `127.0.0.1`; `deploy stop-local`, `deploy start-remote`,
and `deploy stop-remote` provide the corresponding lifecycle operations.

The start and stop commands use the same named-host options:

```bash
uv run run.py deploy start-remote \
  --ray-host plt01=ray-a.example \
  --ray-host plt02=ray-b.example
uv run run.py deploy stop-remote \
  --ray-host plt01=ray-a.example \
  --ray-host plt02=ray-b.example
uv run run.py deploy stop-local
```

SSH uses the current username and configured SSH keys by default. Set
`RAYTEST_SSH_USER` or `RAYTEST_SSH_KEY` to override those settings, or add
`--ssh-password-prompt` to a remote deploy/start/stop command to enter an SSH
password interactively. Credentials are not stored in the inventory.

Each host starts its own Ray head; this does not join the hosts into one Ray cluster.
`--dashboard-host` controls the dashboard bind address only; it is separate
from the advertised node address used for cluster and Ray Client connectivity.

Ray may warn that token authentication is disabled. This is separate from the
uv environment warning. A Ray cluster is trusted-code infrastructure: anyone
who can reach its ports may be able to run code on it. Local-only development
is different from exposing a head on a LAN or the internet; for remote use,
configure Ray authentication and encrypted network access before opening the
Ray ports. See the [Ray token authentication guidance](https://docs.ray.io/en/latest/ray-security/token-auth.html).

For a remote Ray head, the Ray Client port (default `10001`) must be allowed
through the network firewall. The pytest options accept hostnames or IP
addresses; they do not pin a workspace actor to a particular worker.

The configured host selects the Ray cluster, not a specific worker. Ray
schedules the workspace actor on a worker. Worker placement requires Ray
resource requirements or placement strategies, which are outside this example.
The pytest `remote_executor_group_factory` maps friendly names to hostnames or
IP addresses; all executors in one group use the same `RAY_CLIENT_PORT`.

## Project layout

```text
src/sontris/testing/raytest/                        Public executor, pytest API, and plugin
src/sontris/testing/raytest/support/                Reusable execution and Ray host helpers
src/sontris/testing/raytest/fixtures/remote_project/ Packaged remote workload
tests/unittest/                                     Ray-free pytest unit tests
tests/local/                                        Local Ray integration tests
tests/remote/                                       Configured remote Ray integration tests
tests/docker/                                       Docker distribution tests
tests/conftest.py                                   Shared pytest fixtures and host options
scripts/cluster/                                    Local Ray head start/stop helpers
inventory.py                                         Root pyinfra inventory
scripts/pyinfra/                                     Remote pyinfra deployment tasks
pyproject.toml                                      Local package and development dependencies
uv.lock                                             Locked local development environment
```

The packaged fixture project has its own `pyproject.toml` and `uv.lock`; these
are uploaded and used by remote `uv sync` in sontris-raytest's own tests. The fixture also
includes a standard-library-only Rust calculator crate. Its pytest test
uses an existing worker toolchain or installs minimal stable Rust with
rustup, builds the crate remotely, and runs the HTTP API harness. If the
worker lacks a C linker, the setup script uses `apt-get` to install
`build-essential` and `ca-certificates`. This requires a Debian/Ubuntu worker,
root or passwordless sudo, and access to package repositories. Installing
Rust for the first time also requires outbound network access from the worker
to the official rustup and Rust distribution servers.

## Use the executor

The reusable API is available as `sontris.testing.raytest.RemoteExecutor`:

```python
from pathlib import Path

from sontris.testing.raytest import RemoteExecutor

with RemoteExecutor(
    host="127.0.0.1",
    port=10001,
    working_dir=Path.cwd(),
) as executor:
    environment = executor.prepare_environment()
    result = executor.execute_command(
        [environment["python"], "scripts/example.py", "--name", "Ray"]
    )

    if result["timed_out"]:
        raise TimeoutError("Remote command timed out")
    if result["returncode"] != 0:
        raise RuntimeError(result["stderr"])
    print(result["stdout"])
```

When omitted, `host` and `port` default to the `RAY_HOST` and
`RAY_CLIENT_PORT` process environment variables, or to `127.0.0.1` and
`10001`. If `working_dir` is omitted, the executor searches upward from the
current working directory for the nearest `pyproject.toml`.

The default `prepare_environment()` uses `uv sync`. To use another dependency
workflow, pass an argument sequence as `setup_command` and optionally select
the interpreter it creates:

```python
environment = executor.prepare_environment(
    setup_command=["python3", "-m", "venv", ".venv"],
    python_executable=".venv/bin/python",
)
```

To use an already-provisioned interpreter without running setup, pass only
`python_executable`. Paths relative to the workspace are supported. Setup
commands are argument lists and are never run through a shell.

The executor can prepare the remote environment, run a Python file from the
uploaded workspace, execute a system command, download a file or directory,
and inspect workspace/environment information. Its actor-backed workspace
persists for the lifetime of the executor and is removed on close. Use it as
a context manager or call `close()` explicitly. Each executor owns one
workspace actor; Ray chooses the worker. `sontris-raytest` provides remote execution
and pytest integration helpers, not pytest test collection or scheduling
across the cluster.

## License

sontris-raytest is licensed under the GNU Lesser General Public License v3.0 or later
([LGPL-3.0-or-later](LICENCE.md)); the accompanying [COPYING](COPYING) file
contains the GNU GPLv3 text referenced by the LGPL. The LGPL permits use by
proprietary applications, subject to its conditions for distributing the
library or combined works. It does not impose a field-of-use restriction or
require publication of private modifications. It does not grant an exemption
from export-control, sanctions, or other applicable laws.

Third-party dependencies are distributed under their own licenses and are
not relicensed by sontris-raytest. Review each dependency's terms for your use case.

## Development

See [CONTRIBUTING.md](CONTRIBUTING.md) for the architecture, fixture
conventions, and guidance for adapting the executor to another test suite.
