Metadata-Version: 2.4
Name: pytest-horizon
Version: 0.1.0a1
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Rust
Classifier: Topic :: Software Development :: Testing
Requires-Dist: pytest>=8.4,<10
Requires-Dist: pytest-xdist>=3.8 ; extra == 'dev'
Requires-Dist: pytest-playwright>=0.9 ; extra == 'dev'
Requires-Dist: pytest-asyncio>=1 ; extra == 'dev'
Requires-Dist: psutil>=7 ; extra == 'dev'
Provides-Extra: dev
License-File: LICENSE
Summary: Experimental Rust orchestration for ordinary Python pytest workers
Keywords: pytest,parallel,testing,rust
License-Expression: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/MrPuls/pytest-horizon/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/MrPuls/pytest-horizon#readme
Project-URL: Issues, https://github.com/MrPuls/pytest-horizon/issues
Project-URL: Repository, https://github.com/MrPuls/pytest-horizon

# pytest-horizon

An experimental general-purpose parallel execution plugin for pytest. Python runs
pytest, fixtures, hooks, plugins and `conftest.py`. A separate Rust process owns
worker lifecycle, scheduling, deadlines, crash recovery and execution accounting.

The Playwright suite in this repository is a comparison workload. The execution
engine does not import or require Playwright or xdist.

## Install

The `0.1.0a1` release artifacts bundle the Python plugin and Rust controller.
Windows x86-64 and Linux x86-64 wheels install without a Rust compiler. The Linux
wheel requires glibc 2.28 or newer. Python 3.11+ and pytest 8.4–9.x are required;
the initial release was validated on Python 3.14 with pytest 9.1.1.

Once `0.1.0a1` is published to [PyPI](https://pypi.org/project/pytest-horizon/):

```text
python -m pip install pytest-horizon==0.1.0a1
python -m pytest your_tests --horizon=4
```

With uv, use `uv add --dev pytest-horizon==0.1.0a1`, then
`uv run pytest your_tests --horizon=4`. A local wheel can also be installed by
passing its `.whl` path to pip or uv. The alpha is experimental; see the execution
contract and limits below before using it in an existing suite.

Release maintainers: see [the release guide](https://github.com/MrPuls/pytest-horizon/blob/main/docs/RELEASING.md)
for the GitHub Actions workflow and tag instructions.

## Build from source

Source installations require the Rust toolchain. Maturin compiles and installs
the controller as part of the Python package build.

```powershell
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4
```

The plugin locates the controller installed beside its Python interpreter, even
without activating the environment. It also supports local Cargo builds and PATH
as fallbacks. Set `HORIZON_BINARY` or use `--horizon-binary` to provide an explicit
executable. After changing Rust code, reinstall the editable package or point
`HORIZON_BINARY` at the rebuilt Cargo executable.

## Run tests in parallel

```powershell
# Preserve test order and worker affinity within each file (default).
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4 --horizon-schedule=file

# Distribute individual tests for finer balancing.
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4 --horizon-schedule=test

# Phase deadlines apply independently to setup, call and teardown.
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4 --horizon-timeout=30

# Explicitly opt into rerunning attempts interrupted by a worker crash or timeout.
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4 --horizon-retries=1
```

With no `--horizon` option, pytest runs normally. `--collect-only` uses native
pytest collection. Selecting both `--horizon` and xdist `-n` is rejected.

| Option | Default | Purpose |
|---|---|---|
| `--horizon=N` | Disabled | Start N local pytest workers. |
| `--horizon-schedule=file\|test` | `file` | Keep files together or balance individual tests. |
| `--horizon-history=on\|off` | `on` | Use previous durations to schedule longer work first. |
| `--horizon-timeout=SECONDS` | `60` | Deadline for each setup, call and teardown phase. |
| `--horizon-startup-timeout=SECONDS` | `60` | Deadline for worker startup and collection. |
| `--horizon-shutdown-timeout=SECONDS` | `10` | Deadline for worker shutdown. |
| `--horizon-max-restarts=N` | `4` | Maximum worker replacements per run. |
| `--horizon-retries=N` | `0` | Retries for attempts interrupted by a crash or deadline. |
| `--horizon-binary=PATH` | Automatic | Override the bundled Rust executable. |

The `horizon_worker_id` fixture identifies a worker (`hz0`, `hz1`, ...). Use it
to give workers separate external resources when needed. Without parallel mode
it returns `main`. Session fixtures run once per worker, so an existing suite
must tolerate multiple sessions and isolate shared files, accounts or databases.

## Current execution contract

- Every worker runs a real pytest session on its main thread and collects the
  suite independently. Ordered collection manifests must match before dispatch.
- Scheduling owns metadata and indices, never serialized live pytest objects.
  File scheduling preserves collection order inside each file. Optional duration
  history orders scheduling units by longest estimated duration first, with stable
  ties. History is a performance hint, never a reason to skip a test.
- Each command names the current test and an already-reserved next test. The
  worker passes that actual next item into pytest's protocol, retaining fixture
  scopes across commands. Session fixtures belong to each worker session.
- Setup, call and teardown have independent hard deadlines in Rust. Startup /
  collection and shutdown have separate deadlines. A Python faulthandler dump
  provides best-effort stack evidence before a phase deadline expires.
- An interrupted attempt fails by default. The worker's unstarted assignments
  return to the scheduler, and a replacement worker collects and validates the
  manifest before executing more work. Replacement count is bounded.
- Opt-in retries apply only to interrupted attempts, not assertion failures.
  Earlier attempt events remain in the event log and retries are printed.
  Retrying can repeat external side effects; execution is not exactly-once.
- Pytest setup/call/teardown reports are reconstructed in the coordinator. Basic
  reporting, JUnit XML, skips, xfail, fixture errors and pytest-asyncio have
  subprocess integration coverage. Reports are buffered until attempt completion
  so an interrupted retry does not masquerade as an additional final result.
- Global `--maxfail` stops new dispatch; already running tests may finish.
- EOF on the coordinator's control pipe cancels the supervisor. Ctrl+C requests
  cancellation. Windows workers use Job Objects with kill-on-close semantics;
  Linux uses process groups. Both Windows and Linux (Ubuntu under WSL2) have
  subprocess integration coverage. Unix descendants that deliberately leave the
  process group are outside that containment boundary.

## Diagnostics

Each execution writes `.horizon/runs/<run-id>/`:

- `spec.json`: execution settings and pytest arguments.
- `events.jsonl`: timestamped worker and coordinator events.
- `summary.json`: test states, attempt counts, totals, and supervisor exit status.
- `worker-<number>.log`: worker output, native output and best-effort stack dumps.
- `controller.log`: supervisor errors.

IPC uses dedicated handles and bounded event messages (8 MiB). Native writes to
stdout are redirected to worker logs so test output cannot corrupt the protocol.
Rust serializes each outgoing event once and writes a complete JSON frame to the
log and reporting pipe, avoiding a filesystem write for each formatted field.
Test output under `-s` is retained in those logs; live multiplexed output is not
implemented. Duration history is stored in `.horizon/durations.json`.

## Compare with xdist

The repository includes 48 local Playwright tests across six files: forms, delayed
DOM updates, intercepted API requests, isolated storage, navigation and a
deliberately slower file. No remote website, account or server is required.

```powershell
$env:PLAYWRIGHT_BROWSERS_PATH = Join-Path (Get-Location) '.tools\browsers'
.\.venv\Scripts\python.exe -m playwright install chromium --only-shell
.\.venv\Scripts\python.exe scripts/compare.py --workers 4 --repeats 3 --faults
```

The comparison alternates engines, uses fresh processes and browsers, records
complete command wall time, and checks per-test phase outcomes. It compares file
scheduling with xdist `loadfile`, and individual scheduling with xdist `load`.
Horizon runs both with and without populated historical durations. Every variant
gets an unmeasured priming run. Results include every command and raw report.

The exact dependency versions for the recorded comparison are pinned in
`requirements-benchmark.txt`. A tracked [validation summary](https://github.com/MrPuls/pytest-horizon/blob/main/docs/VALIDATION.md)
records the tested environments and outcomes. Each comparison generates a local
`artifacts/comparison-*/REPORT.md` and `results.json`; raw logs and generated
artifacts are excluded from Git.

Separate fault runs deliberately hang or kill a worker after a browser starts.
Horizon gets a 3-second phase deadline; a common 12-second external watchdog bounds
both engines. This tests default xdist without an additional timeout plugin. It
does not reproduce or explain the original reported xdist freeze.

## Verification

```powershell
cargo test --locked
cargo build --release --locked
.\.venv\Scripts\python.exe -m pip install -r requirements-compatibility.txt
$env:HORIZON_BINARY = Join-Path (Get-Location) 'target\release\horizon-controller.exe'
$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD = '1'
.\.venv\Scripts\python.exe -m pytest tests -p pytest_asyncio.plugin -q
Remove-Item Env:PYTEST_DISABLE_PLUGIN_AUTOLOAD
```

The tests start real Rust supervisors and Python workers. Plugin dependencies are
loaded explicitly in each subprocess; a separate case tests installed plugins
together with normal auto-loading. Missing optional plugin dependencies produce
skips, so install the compatibility requirements before claiming matrix coverage.

The expanded suite exercises pytest-cov (branch aggregation, test contexts,
early conftest imports and failure thresholds), pytest-randomly (shared seeds),
pytest-mock, pytest-html (worker extras), pytest-rerunfailures (including
`--fail-on-flaky`), pytest-timeout, Hypothesis, AnyIO and pytest-asyncio.
Adapters in `python/pytest_horizon/compatibility.py` use some plugin internals;
the pinned versions are the verified contract, not all past or future releases.
Coverage from a forcibly killed worker can be incomplete.
For `--cov-context=test`, use `COVERAGE_CORE=ctrace` or `[run] core=ctrace` in
your coverage configuration. With the tested coverage version on Python 3.14,
the default sys.monitoring tracer produced missing/mislabelled contexts in
plain pytest, xdist and Horizon. Horizon rejects that unsupported combination
before starting workers instead of publishing misleading per-test attribution.

For longer browser validation:

```powershell
.\.venv\Scripts\python.exe scripts/stress.py --output artifacts/stress-local
```

This starts 12 paired cycles using one to eight workers, a 960-test session per
engine, repeated Horizon browser crash/hang recovery, and bounded xdist failure
cases with and without pytest-timeout. Each run retains commands, reports,
measurements and Horizon state summaries. This is a correctness stress check;
the original comparison harness is better suited to controlled timing runs.

For Linux, install Python venv support, a C linker and Cargo, then run
`bash scripts/setup_linux.sh`. If required, install Chromium's system libraries
using Playwright's `install-deps chromium` command with system permissions.

```bash
export HORIZON_BINARY="$PWD/target/linux/release/horizon-controller"
export PLAYWRIGHT_BROWSERS_PATH="$PWD/.tools/linux/browsers"
export PYTHONPYCACHEPREFIX="$PWD/.tools/linux/pycache"
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .tools/linux/venv/bin/python -m pytest tests \
  -p pytest_asyncio.plugin -q -o cache_dir=.tools/linux/pytest-cache
.tools/linux/venv/bin/python scripts/stress.py --output artifacts/stress-linux
```

Use separate bytecode, pytest cache and browser output paths when Windows and WSL
share this checkout. WSL uses a Linux binary and Linux browser; measurements from
a mounted Windows filesystem should not be treated as native Linux performance.
The recorded hardening results are summarized in
[docs/VALIDATION.md](https://github.com/MrPuls/pytest-horizon/blob/main/docs/VALIDATION.md).
Detailed reports and machine-readable data are generated locally under `artifacts/`.

## Prototype limits

This is not a full xdist replacement or a promise of compatibility with every
pytest plugin. There is no remote execution, live debugging, resource-lock API,
custom scheduler hook API, warning forwarding, or full compatibility with
xdist-specific hooks. Arbitrary plugin-defined report payloads may require
additional JSON normalization.

The coordinator is still a Python pytest process: blocking coordinator hooks or
blocked local storage can stall reporting. The independent Rust phase watchdog
does not make every possible coordinator or operating-system failure bounded.
Forced termination cannot guarantee fixture finalizers or finalized browser
traces. Different valid schedules can expose shared-state and order-dependent
tests. Historical scheduling makes decisions consistent for a given manifest and
history; it does not promise identical concurrent execution order.

Broader plugin and platform compatibility needs further work. The current evidence
is a local synthetic comparison and deliberately injected failures, not a general
performance or reliability claim.

## Report an issue

Open an [issue](https://github.com/MrPuls/pytest-horizon/issues) with your OS,
Python, pytest and plugin versions, the command used, and a minimal reproducer.
For a hang or crash, include the relevant Horizon summary and worker log after
removing credentials and application data. Logs can contain test output and
pytest arguments.

See the [changelog](https://github.com/MrPuls/pytest-horizon/blob/main/CHANGELOG.md)
for release notes. Licensed under the [MIT license](https://github.com/MrPuls/pytest-horizon/blob/main/LICENSE).

