# Testenix complete documentation

> Generated from the Testenix repository. Treat current 0.1 behavior separately from roadmap items, and treat every benchmark as workload-specific.

Canonical documentation: https://polishdataengineer.github.io/testenix/
Source repository: https://github.com/polishdataengineer/testenix

---

# Document: Overview

Canonical URL: https://polishdataengineer.github.io/testenix/
Source: docs/index.md

---
description: Testenix is an async-native, parallel-first Python testing framework with lossless results.
---

# Testenix

<div class="testenix-hero">
  <div class="testenix-kicker">Python testing framework · Alpha</div>
  <div class="testenix-title">Fast tests. Clear results.</div>
  <p>
    Testenix combines a dependency-free native runtime with a transparent bridge for
    running existing pytest suites unchanged.
  </p>
  <div class="testenix-actions">
    <a href="https://polishdataengineer.github.io/testenix/getting-started/">Start testing</a>
    <a href="https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/">Run pytest suites</a>
    <a href="https://polishdataengineer.github.io/testenix/guides/migration/">Migrate safely</a>
    <a href="https://polishdataengineer.github.io/testenix/benchmarks/results/">See the benchmarks</a>
    <a href="https://polishdataengineer.github.io/testenix/for-llms/">Copy docs for an LLM</a>
  </div>
</div>

```{toctree}
:hidden:
:maxdepth: 2

getting-started
guides/pytest-compatibility
guides/migration
guides/writing-tests
guides/fixtures
guides/parallelism
guides/reports
reference/cli
reference/configuration
reference/api
benchmarks/results
benchmarking
performance-analysis
architecture
roadmap
for-llms
```

## Why Testenix

<div class="metric-grid">
  <div class="metric-card"><strong>0</strong><span>dependencies in the native runtime</span></div>
  <div class="metric-card"><strong>12</strong><span>Python and OS combinations in CI</span></div>
  <div class="metric-card"><strong>3</strong><span>console, JSON, and JUnit reports</span></div>
  <div class="metric-card"><strong>3.15×</strong><span>native <code>testenix run</code> on the 100k synthetic workload vs pytest</span></div>
</div>

Testenix is deliberately built around a few strong guarantees:

- **Async is native.** Coroutine tests and async-generator fixtures use the same model as
  synchronous code and do not require a plugin.
- **Parallelism is part of the runner.** Module affinity, process isolation, and
  duration-aware scheduling are designed together.
- **Retries preserve evidence.** A failed attempt followed by a pass is `FLAKY`, never silently
  rewritten as a clean pass.
- **Crashes cannot erase completed work.** Workers stream results as tests finish, and unfinished
  tests receive explicit terminal outcomes.
- **Reports share one model.** Console, JSON, JUnit, history, and the library API are derived from
  the same versioned result contracts.

## A complete first test

Already have a pytest suite? Keep its fixtures, parametrization, markers, classes, configuration,
and plugins:

```console
$ python -m pip install "testenix[pytest]"
$ testenix pytest -q tests
```

[Read the compatibility contract](https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/) before migrating individual
modules to the native engine.

To create a validated native copy without modifying the originals, use:

```console
$ testenix migrate auto tests --dry-run
$ testenix migrate auto tests --check
$ testenix migrate auto tests --output tests_testenix
```

The migrator executes the source baseline and both serial and parallel native candidates in
disposable project copies, compares their inventories and outcomes, and publishes only through an
atomic no-overwrite rename. [Read the safe migration contract](https://polishdataengineer.github.io/testenix/guides/migration/).

```python
from collections.abc import AsyncIterator

from testenix import case, cases, fixture, test


@fixture(scope="module")
async def multiplier() -> AsyncIterator[int]:
    yield 2


@test("multiplication uses an async fixture", tags={"unit"})
@cases(
    case(id="positive", value=3, expected=6),
    case(id="zero", value=0, expected=0),
)
async def multiplication(multiplier: int, value: int, expected: int) -> None:
    assert multiplier * value == expected
```

Run it locally:

```console
$ python -m pip install testenix
$ testenix run tests
```

If the first PyPI release is not available yet, install the current source with
`python -m pip install "testenix @ git+https://github.com/polishdataengineer/testenix.git@main"`.

## Performance evidence, with context

The checked-in development baseline measured native `testenix run` on 100,000 empty tests across
16 generated modules on an Apple M4 Pro and CPython 3.11. Native Testenix completed that specific
workload in a median 8.04 seconds, compared with 25.33 seconds for pytest and 21.30 seconds for
pytest-xdist. These measurements do not apply to the delegated `testenix pytest` command.

<div class="benchmark-caveat">
This is a preliminary synthetic result from one machine, not a promise that every project will be
3.15× faster. The benchmark page publishes the raw samples, environment, variance, methodology,
and limitations so that the claim can be evaluated rather than taken on trust.
</div>

[Inspect the benchmark data](https://polishdataengineer.github.io/testenix/benchmarks/results/) or
[reproduce the harness](https://polishdataengineer.github.io/testenix/benchmarking/).

## Project maturity

Testenix is alpha software. The `testenix pytest` bridge preserves an existing suite by delegating
to real pytest, while `testenix run` is a distinct native engine rather than a drop-in pytest
reimplementation. Pytest still has a much broader plugin ecosystem, richer IDE integration, and
mature assertion rewriting. Choose the bridge for compatibility and the native engine when its
async model, built-in parallel execution, explicit failure semantics, or dependency-free core are
more important.

---

# Document: Getting started

Canonical URL: https://polishdataengineer.github.io/testenix/getting-started/
Source: docs/getting-started.md

# Getting started

This guide takes a new project from installation to its first parallel Testenix run.

## Requirements

- CPython 3.11 or newer
- Linux, macOS, or Windows
- no runtime dependencies beyond the Python standard library for native `testenix run`
- pytest in the same environment when using the optional compatibility bridge

## Install

Install the released package from PyPI:

```console
$ python -m pip install testenix
$ testenix --version
```

For a project managed by uv:

```console
$ uv add --dev testenix
$ uv run testenix --version
```

For an existing pytest project, install the compatibility extra:

```console
$ python -m pip install "testenix[pytest]"
# or
$ uv add --dev "testenix[pytest]"
```

Until the first PyPI release is visible, install directly from the protected `main` branch:

```console
$ python -m pip install "testenix @ git+https://github.com/polishdataengineer/testenix.git@main"
# include pytest when the project environment does not already provide it
$ python -m pip install "testenix[pytest] @ git+https://github.com/polishdataengineer/testenix.git@main"
```

## Choose an execution mode

Run an unchanged pytest suite through its real engine:

```console
$ testenix pytest -q tests
```

Use `testenix run` for native Testenix tests and the built-in scheduler, retries, history, and
lossless reports. The two commands have deliberately separate semantics. See
[pytest compatibility](https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/) for the capability matrix and migration
boundary.

Create a validated native copy of supported pytest or unittest modules without changing the
originals:

```console
$ testenix migrate auto tests --dry-run
$ testenix migrate auto tests --output tests_testenix
$ testenix run tests_testenix
```

Migration requires a green source run and matching serial/parallel native runs before an atomic
new-directory publish. Read [safe migration](https://polishdataengineer.github.io/testenix/guides/migration/) before using it on a suite with
external side effects.

## Create a native test

Testenix collects ordinary functions whose names begin with `test_`. Decorators are optional for
simple tests.

```python
# tests/test_math.py

def test_addition() -> None:
    assert 2 + 2 == 4
```

Run the suite:

```console
$ testenix run tests
```

The process exits with code `0` when every selected test has a non-gating terminal status.

## Add native metadata

Use `@test` when a description, tags, or a hard timeout should be part of the test contract.

```python
from testenix import test


@test("addition remains correct", tags={"unit", "fast"}, timeout=2.0)
def addition() -> None:
    assert 2 + 2 == 4
```

Select tagged tests by repeating `--tag`. Repeated tags use AND semantics:

```console
$ testenix run --tag unit --tag fast
```

## Configure the project

Put stable defaults in `pyproject.toml`:

```toml
[tool.testenix]
paths = ["tests"]
workers = "auto"
retries = 0
history = ".testenix/history.sqlite3"
# timeout = 10
# json = "reports/testenix.json"
# junit = "reports/junit.xml"
```

Command-line values override the project table:

```console
$ testenix run --workers 4 --retries 1 --json reports/result.json
```

Use `--no-history` for a side-effect-free run. Duration history normally helps later runs schedule
long tests earlier.

## Use it from Python

The runner also exposes a typed API:

```python
from testenix import Status, TestenixConfig, run

result = run("tests", TestenixConfig(workers=4, history_path=None))
failed = [item.test.id for item in result.tests if item.status is not Status.PASS]
```

Async applications can call `await testenix.run_async(...)`. Cancellation terminates active
collection and execution process trees before control returns to the caller.

## Next steps

- [Write tests, cases, tags, skips, and expected failures](https://polishdataengineer.github.io/testenix/guides/writing-tests/)
- [Run or migrate an existing pytest suite](https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/)
- [Convert pytest or unittest safely without replacing originals](https://polishdataengineer.github.io/testenix/guides/migration/)
- [Build fixture graphs](https://polishdataengineer.github.io/testenix/guides/fixtures/)
- [Understand process parallelism and timeouts](https://polishdataengineer.github.io/testenix/guides/parallelism/)
- [Produce JSON and JUnit reports](https://polishdataengineer.github.io/testenix/guides/reports/)
- [Review every configuration option](https://polishdataengineer.github.io/testenix/reference/configuration/)

---

# Document: Pytest compatibility

Canonical URL: https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/
Source: docs/guides/pytest-compatibility.md

---
description: Run existing pytest suites unchanged through Testenix and understand the native migration boundary.
---

# Pytest compatibility

Testenix can run an existing pytest suite without rewriting it. Use the compatibility bridge when
the suite depends on pytest fixtures, parametrization, markers, classes, plugins, hooks, or
configuration:

```console
$ python -m pip install "testenix[pytest]"
$ testenix pytest -q tests
```

For the supported static subset, `testenix migrate pytest tests` can instead create a validated
native copy without modifying the source. [The safe migration guide](https://polishdataengineer.github.io/testenix/guides/migration/) defines its
strict support and rollback contract.

If a supported pytest (`>=8.3,<10`) is already installed in the same Python environment, installing
the base `testenix` package is enough. The extra is a convenience for environments that do not
already contain pytest.

## What the command does

Everything after `testenix pytest` is forwarded unchanged to:

```console
$ python -m pytest [PYTEST_ARGS ...]
```

Testenix hands its current CLI process to pytest from the same Python interpreter. On POSIX it uses
a process overlay; on Windows it calls pytest's public console entry point in-process because the
platform does not provide the same overlay semantics. Both paths preserve the foreground process,
working directory, environment, terminal, standard streams, and pytest's signal handling. Pytest
therefore remains responsible for collection, execution, configuration, plugin loading, output,
descendants such as pytest-xdist workers, and exit status.

```console
$ testenix pytest tests/test_api.py::test_health -k smoke --maxfail=1
$ testenix pytest -m "unit and not slow" tests
$ testenix pytest -n auto tests
$ testenix pytest --junitxml=reports/pytest.xml tests
```

The `-n` example requires pytest-xdist in the same environment. Testenix does not translate its
native `--workers` option into an xdist option.

## Capability matrix

| Capability | `testenix pytest` | `migrate` then `run` | Direct `testenix run` |
| --- | --- | --- | --- |
| Plain module-level pytest-default `test*` functions | Yes, through pytest | Yes | `test_` or native `@test` |
| Pytest classes | Yes | Blocked | No |
| Simple local fixture | Yes | Converted | Only native `@fixture` |
| Session-scoped pytest fixture | Yes | Blocked: run-global vs worker-local | Native scope is worker-local |
| Built-in/dynamic fixture | Yes | Blocked | No pytest fixture semantics |
| Adjacent `conftest.py` fixture | Yes | Simple static subset | No automatic conftest discovery |
| Static single `parametrize` | Yes | Converted to cases | Use `@case` or `@cases` |
| Skip and plain custom marker | Yes | Converted | Use Testenix decorators/tags |
| Pytest xfail | Yes | Blocked due semantic differences | Use native `@xfail` intentionally |
| Pytest assertion rewriting | Yes | No | No |
| Plugins, hooks, pytest config | Yes | Blocked/not translated | No |
| Async plugin semantics | According to plugins | Blocked | Native async needs no plugin |
| Testenix worker scheduler/history | No | Yes after migration | Yes |
| Testenix retries and lossless results | No | Yes after migration | Yes |
| Published native speedups | No | Measure generated suite | Only documented workloads |
| Source preservation | Runs source | Source stays untouched | Not applicable |

Some simple pytest-authored functions also happen to run with `testenix run`, but that overlap is
not a compatibility guarantee. In particular, the native collector does not interpret pytest
markers. Running a pytest suite through the native command can turn a skip or xfail into an
ordinary execution, collapse parametrized cases, miss class-based tests, or fail fixture setup.
Use the explicit compatibility command until a suite has been intentionally migrated.

## Boundaries

`testenix pytest` does not use `[tool.testenix]`, the native collector, worker pool, retries,
timeouts, tags, history, event model, JSON reporter, or JUnit reporter. Pass the corresponding
pytest or plugin options after the subcommand. For example, use pytest's `--junitxml`, not
Testenix's native `--junit`.

Pytest and every required plugin must be installed beside the `testenix` executable in the same
interpreter environment. For uv-managed projects, prefer:

```console
$ uv add --dev "testenix[pytest]"
$ uv run testenix pytest -q tests
```

An isolated `uv tool install testenix` environment does not automatically see pytest plugins from
the project environment. Install the required packages into the tool environment or invoke
Testenix through the project environment instead.

After the handoff, pytest owns signal handling and returns any status chosen by pytest or a plugin,
unchanged. If pytest is missing, Testenix returns `2` with an installation hint before the handoff.
Failure to hand the process to pytest returns `3`.

## Performance and benchmark interpretation

Compatibility mode has pytest's execution performance plus launcher and adapter overhead, which has
not yet been measured separately. It is not expected to be faster than invoking `python -m pytest`
directly. The published Testenix benchmarks exercise the native `testenix run` engine and must not
be used to describe `testenix pytest`.

## Migration path

Use the bridge first to put Testenix in front of an unchanged suite without altering its semantics.
Then let the conservative migrator identify and validate modules whose pytest dependencies have
explicit equivalents:

1. keep the whole suite green with `testenix pytest`;
2. run `testenix migrate pytest tests --dry-run` and inspect every diagnostic;
3. use `--check --report-json ...` to execute the source, serial native, and parallel native gates;
4. publish to a new directory only after all three inventories/outcomes agree;
5. keep unsupported modules on `testenix pytest` and keep all originals during rollout;
6. benchmark the generated directory before making a native performance claim.

The migrator replaces simple fixtures, parametrization, skip conditions, and plain markers in the
generated copy. It never performs edits in place. Manual rewrites are still necessary for blocked
plugin, hook, built-in fixture, class, xfail, and dynamic behavior.

The current bridge does not convert pytest outcomes into a Testenix `RunResult`. Deeper event and
report aggregation is a future compatibility layer and requires explicit pytest hook integration.

---

# Document: Safe migration

Canonical URL: https://polishdataengineer.github.io/testenix/guides/migration/
Source: docs/guides/migration.md

---
description: Safely convert supported pytest and unittest suites to native Testenix without replacing the originals.
---

# Safe migration to native Testenix

An existing suite can use Testenix in three ways:

| Goal | Command | Execution engine | Original suite |
| --- | --- | --- | --- |
| Preserve all pytest behavior | `testenix pytest -q tests` | pytest | Runs directly |
| Check whether native conversion is supported | `testenix migrate auto tests --dry-run` | no tests run | Read-only |
| Create a validated native copy | `testenix migrate auto tests -o tests_testenix` | Testenix after conversion | Kept and used as the baseline |

Migration is conservative. It stops on a construct whose behavior cannot be proven equivalent
instead of producing a plausible-looking, incomplete suite.

## First migration

For pytest, install the optional dependency because validation executes the real source suite:

```console
$ python -m pip install "testenix[pytest]"
$ testenix migrate pytest tests --dry-run
$ testenix migrate pytest tests --check --workers 2 \
    --report-json reports/migration-check.json
$ testenix migrate pytest tests --output tests_testenix \
    --report-json reports/migration.json
$ testenix run tests_testenix
```

For a standard-library unittest suite, no extra package is needed:

```console
$ testenix migrate unittest tests --output tests_testenix
```

Use `auto` when pytest-style modules and unittest modules live in the same selected directory.
They must be separate modules; a file that mixes both test models is rejected.

The destination defaults to `testenix_migrated`. It must be a new directory inside the project,
with an existing real parent. There is deliberately no `--force` and no in-place mode.
An integer `--workers` value must be at least 2 so the parallel gate cannot silently repeat the
serial command. A one-module suite still has one schedulable affinity unit, which is disclosed as a
`MIG006` warning; spread tests across independent modules to exercise multiple workers. An
audit-report path must also be new, inside the project, and disjoint from every selected source and
the output suite. Testenix never replaces an existing report.

## Transaction and rollback contract

Testenix uses copy-and-validate rather than edit-and-undo:

```text
preflight paths
  -> snapshot source names + SHA-256
  -> convert in memory
  -> run original suite in disposable project copy
  -> run generated suite with one Testenix worker in a fresh copy
  -> run generated suite in parallel in another fresh copy
  -> compare inventory, totals, and every mapped test outcome
  -> recheck every source hash
  -> atomically publish a new directory without replacement
```

If parsing, conversion, the source baseline, either native run, parity comparison, a source hash,
or publication fails before the atomic rename, the requested output directory remains absent. A
failure to write the optional report after a successful rename instead leaves the validated output
published, prints an explicit warning, and returns the successful migration status. Testenix never writes to,
renames, deletes, or replaces a selected source file. There are therefore no old files to restore:
the old suite was never changed.

`--dry-run` stops after static analysis and a final source-hash check. `--check` performs the full
three-run validation but does not publish the destination. The default performs the same checks
and then uses an operating-system no-replace rename. If another process creates the destination
during validation, publication fails closed.

POSIX staging creation, artifact writes, publication, and cleanup are anchored to open directory
descriptors. On a platform without safe descriptor-relative recursive deletion, a non-empty
failed staging transaction is retained under `.testenix/migrations` for inspection instead of
being removed with a path-based recursive walk.

Validation runs with the shadow as both the current directory and `PWD`, so ordinary relative-path
writes land in the disposable copy. A shadow is not an operating-system security sandbox: a test
can still write through a hard-coded absolute path or another environment variable, and network,
database, cloud, and subprocess side effects remain external. Selected Python sources are rehashed
before every terminal result, so detected drift stops publication, but Testenix never tries to undo
an independent user or test-process edit. Use test credentials and `--dry-run` first for suites
with external effects.

## Pytest conversion contract

The current converter supports the subset below:

- module-level pytest-default `test*` functions and normal Python `assert` statements;
- one static `pytest.mark.parametrize` with static names, rows, IDs, and unmarked
  `pytest.param(..., id=...)` values;
- local fixtures with no parameters, autouse, or `request`, using function or module scope;
- simple fixtures from an adjacent `conftest.py` in the same directory;
- static `pytest.mark.skip` and `pytest.mark.skipif`;
- plain argument-free custom markers, converted to Testenix tags;
- pytest runtime helpers `approx`, `deprecated_call`, `fail`, `raises`, and `warns`. Generated
  modules using these helpers still require pytest at runtime.

It blocks, with a file and line diagnostic:

- pytest test classes, xfail, runtime skip/xfail/importorskip/exit, xunit lifecycle hooks;
- built-in fixtures such as `tmp_path`, `monkeypatch`, `capsys`, and `request`;
- autouse or parametrized fixtures, fixture overrides, and inherited ancestor-`conftest` fixtures;
- session-scoped fixtures, because pytest creates one per run while Testenix session scope is
  currently worker-local;
- stacked, dynamic, indirect, scoped, or per-case-marked parametrization;
- `usefixtures`, module-level `pytestmark`, hook functions, plugin registration, and semantic
  plugin markers such as asyncio/anyio, timeout, order, repeat, or flaky;
- decorators and required parameters whose execution meaning cannot be established statically.

Any converted pytest file whose name is not already `test_*.py` is renamed in the generated copy
to a native-discoverable name. Explicit nonstandard files such as `specs.py` are supported when
they contain static test inventory. Supporting Python modules and package `__init__.py` files under
the selected source directories are copied to preserve common relative imports. Missing non-Python
assets or path-sensitive `__file__` behavior will make candidate validation fail rather than
publish a bad copy.

Pytest plugins are intentionally not emulated. Keep using `testenix pytest` for modules that need
them and migrate supported modules incrementally.

## Unittest conversion contract

Rewriting `self.assertEqual`, lifecycle methods, cleanup stacks, and mock decorators would be
fragile. Instead, Testenix generates one native function wrapper per test method. The wrapper:

1. resolves the original by the exact wrapper-to-project relative path, independent of `cwd`;
2. verifies a manifest containing every selected Python source and SHA-256;
3. loads the original class by exact path;
4. executes one method through the standard `TestCase.run(TestResult)` protocol;
5. exposes its pass, failure, skip, expected-failure, or unexpected-success outcome to Testenix.

This preserves direct subclasses of `unittest.TestCase` and
`unittest.IsolatedAsyncioTestCase`, including per-test `setUp`/`tearDown`, async lifecycle,
`addCleanup`, `assert*`, context-manager assertions, and `unittest.mock.patch`. Static skips map to
Testenix `SKIP`; `expectedFailure` maps to `XFAIL`; an unexpected success remains the gating
`XPASS` outcome.

The generated unittest suite deliberately depends on the unchanged originals. Do not delete or
move them, and do not move or rename the published generated directory: wrappers encode the exact
relative relationship between both trees. If either path or an original file changes, rerun
migration; a changed source makes native collection fail with an explicit diagnostic.

The converter blocks `subTest`, runtime `skipTest`/`SkipTest`, class/module lifecycle and cleanup,
custom `run` or loader hooks, `load_tests`, mixins or indirect inheritance, `FunctionTestCase`,
dynamic/DDT/parameterized method generation, metaclasses, and ambiguous decorators.

## What validation compares

The source suite must be green. Testenix records and compares:

- the exact collected test count against the converter's source-to-target mapping;
- passed, failed, error, skipped, expected-failure, and unexpected-success totals;
- the outcome of every source test against its exact generated target mapping;
- native serial and native parallel outcome signatures;
- every selected Python source path and SHA-256 immediately before publication.

A pre-existing source failure is not treated as proof of equivalent conversion, even when the
candidate happens to fail too. Fix the baseline or use `--dry-run` to inspect static support.

The JSON audit report uses the versioned `testenix.migration-report` format. It includes source
hashes, every source-to-target mapping, per-test outcomes, generated files, line diagnostics,
timings and summaries for all validation runs, publication status, and an `originals_modified`
flag. It is false for a successful transaction and true when a terminal source recheck detects
drift; the flag reports observed state and does not claim that Testenix caused an independent edit.

## Performance with thousands of migrated tests

Migration unlocks the native scheduler, process supervision, async engine, retries, history, and
Testenix reports. It does not guarantee that every converted suite is faster. Testenix keeps a
normal module as one affinity unit so module-scoped fixtures are not duplicated. Consequently,
3,000 tests in one source module still form one schedulable unit; 3,000 tests spread across enough
independent modules can use multiple workers.

The checked-in migration baseline used an Apple M4 Pro, CPython 3.11, 3,000 generated tests in 64
modules, four native workers, one warm-up, and five measured rounds:

| Source runner | Test body | Source median | Native median | Native result | One-time migration |
| --- | --- | ---: | ---: | ---: | ---: |
| sequential pytest | no-op | 1.539 s | 0.521 s | 2.96x faster | 5.940 s |
| sequential unittest outcome probe | no-op | 0.161 s | 1.192 s | 7.40x slower | 6.742 s |
| sequential unittest outcome probe | 1 ms sleep | 4.066 s | 2.577 s | 1.58x faster | 17.251 s |

The migration column is the complete copy, source-baseline, serial-candidate,
parallel-candidate, integrity-check, and publication transaction. It is paid when regenerating a
suite and is deliberately excluded from the recurring execution medians. The unittest adapter
loads the unchanged source and translates `TestCase.run()` results for every native wrapper. That
fixed cost dominates an empty method; once the generated methods contain 1 ms of synthetic work,
parallel execution across 64 modules outweighs it in this scenario.

These source measurements use sequential pytest and a sequential Testenix probe built on the
stdlib unittest loader/result protocol. The probe serializes per-test outcomes for the parity gate,
so its small audit overhead is included. They are not comparisons
against pytest-xdist or a third-party parallel unittest runner. They are synthetic timing evidence,
not a promise for a real project: imports, fixtures, I/O, test-duration distribution, module
affinity, worker count, operating system, and CPU can all reverse the result. See the
[generated benchmark page](https://polishdataengineer.github.io/testenix/benchmarks/results/) for raw samples and variance, or inspect the
checked-in [pytest](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_pytest_3000.json),
[unittest no-op](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_unittest_3000.json),
and [unittest 1 ms](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_unittest_3000_delay_1ms.json)
JSON files directly.

Benchmark the generated directory only after parity validation:

```console
$ python benchmarks/run_migration_benchmark.py \
    --framework pytest --tests 3000 --modules 64 --workers 4 \
    --warmups 1 --repeats 5 --output migration-pytest-3000.json
$ python benchmarks/run_migration_benchmark.py \
    --framework unittest --tests 3000 --modules 64 --workers 4 \
    --warmups 1 --repeats 5 --output migration-unittest-3000.json
$ python benchmarks/run_migration_benchmark.py \
    --framework unittest --tests 3000 --modules 64 --workers 4 --delay-ms 1 \
    --warmups 1 --repeats 5 --output migration-unittest-3000-delay-1ms.json
```

The harness checks the exact count and source hashes before accepting any timing sample. Publish
the raw JSON, environment, module count, worker count, variance, and commands with a result. Do not
apply the repository's synthetic native benchmark ratio to a real migrated project without
measuring it.

## CI rollout

A low-risk rollout keeps both suites for several releases and runs the exact newly published copy:

```yaml
- name: Generate and validate a fresh native copy
  run: |
    testenix migrate auto tests --output tests_testenix_ci --workers 2 \
      --report-json migration-report.json

- name: Existing source suite
  run: testenix pytest -q tests

- name: Run the exact validated copy
  run: testenix run tests_testenix_ci --no-history
```

The output and report paths must be absent at job start, which is natural on a fresh CI checkout.
`--check` validates an ephemeral candidate; it does not prove that a separately committed
`tests_testenix` directory has identical bytes. Regenerate and publish a fresh CI output as above,
or add a project-specific content-manifest check for a committed copy. Regenerate whenever a pinned
unittest source changes.

## Exit codes

| Code | Migration meaning |
| ---: | --- |
| `0` | Analysis, check, or publication completed successfully. |
| `1` | Source or candidate execution failed, timed out, or produced different outcomes. |
| `2` | Unsafe/invalid path, existing output, source drift, or command usage error. |
| `3` | Internal process or pre-publication report failure. A report-only failure after publication warns and preserves exit `0`. |
| `4` | At least one construct is unsupported by the conservative converter. |
| `130` | Interrupted by the user. |

Diagnostics beginning with `PYT` describe pytest source, `UNIT` describes unittest source, and
`MIG` describes cross-framework inventory or transaction safety.

---

# Document: Writing tests

Canonical URL: https://polishdataengineer.github.io/testenix/guides/writing-tests/
Source: docs/guides/writing-tests.md

# Writing tests

Testenix tests are ordinary typed functions. Discovery recognizes `test_*` names, while decorators
add explicit metadata without wrapping or changing the callable.

## Plain functions

```python
def test_total() -> None:
    assert sum([2, 3, 5]) == 10
```

Both synchronous functions and coroutines are supported:

```python
async def test_async_client() -> None:
    response = await fetch_record(42)
    assert response["id"] == 42
```

## Descriptions, tags, and timeouts

```python
from testenix import test


@test(
    "the cache expires stale entries",
    tags={"unit", "cache"},
    timeout=1.5,
)
async def cache_expiry() -> None:
    ...
```

The decorator attaches immutable metadata and preserves the original function signature,
annotations, and traceback.

A timeout is a hard process deadline. This is more expensive than shared-worker execution, but it
can stop both a blocked coroutine and a blocking synchronous call.

## Parameter cases

Use `case` for named examples:

```python
from testenix import case, cases, test


@test("discount rules")
@cases(
    case("regular", amount=100, rate=0.0, expected=100),
    case("member", amount=100, rate=0.1, expected=90),
)
def discount(amount: int, rate: float, expected: float) -> None:
    assert amount * (1 - rate) == expected
```

Use keyword dimensions to create a Cartesian product:

```python
@cases(
    role=["admin", "editor"],
    active=[True, False],
)
def test_permissions(role: str, active: bool) -> None:
    ...
```

Case values are rebuilt when the worker rediscovers the module, so they do not need to cross the
process boundary through pickle. They must still be reproducible during module import.

## Skip and expected failure

```python
import sys

from testenix import skip, xfail


@skip("Windows-only behavior", when=sys.platform != "win32")
def test_windows_registry() -> None:
    ...


@xfail("tracked as issue #42")
def test_known_edge_case() -> None:
    assert current_behavior() == desired_behavior()
```

An expected failure becomes `XFAIL`. If the test unexpectedly passes, the result is `XPASS` and
gates the run.

## Retries and flakiness

Retries can be configured globally or on the command line:

```console
$ testenix run --retries 2
```

Every attempt remains in the result model. `FAIL -> PASS` is finalized as `FLAKY` and returns a
gating exit code; Testenix never hides the initial failure.

## Tags

Tags are normalized strings stored in the test specification:

```python
@test(tags={"integration", "database"})
def database_round_trip() -> None:
    ...
```

Run tests containing every requested tag:

```console
$ testenix run --tag integration --tag database
```

An explicit tag filter that selects no tests is treated as a usage error and exits with code `2`.

---

# Document: Fixtures

Canonical URL: https://polishdataengineer.github.io/testenix/guides/fixtures/
Source: docs/guides/fixtures.md

# Fixtures

Fixtures are typed dependency providers. A test requests a fixture by using its name as a function
parameter.

## Return-value fixtures

```python
from testenix import fixture


@fixture
def customer_id() -> int:
    return 42


def test_customer(customer_id: int) -> None:
    assert customer_id > 0
```

## Cleanup with generators

Synchronous and asynchronous generator fixtures run cleanup after the consumer:

```python
from collections.abc import AsyncIterator

from testenix import fixture


@fixture
async def client() -> AsyncIterator[Client]:
    value = await Client.connect()
    try:
        yield value
    finally:
        await value.close()
```

Setup, call, and teardown are separate result phases. A teardown failure remains visible even when
the test body passed.

## Fixture dependencies

Fixtures can request other fixtures:

```python
@fixture
def database_url() -> str:
    return "sqlite:///:memory:"


@fixture
def repository(database_url: str) -> Repository:
    return Repository(database_url)


def test_empty_repository(repository: Repository) -> None:
    assert repository.count() == 0
```

The dependency graph is validated before execution. Missing fixtures and cycles become collection
issues instead of hanging the run.

## Scopes

```python
@fixture(scope="module")
def module_resource() -> Resource:
    return Resource()


@fixture(scope="session")
def worker_resource() -> Resource:
    return Resource()
```

| Scope | Lifetime in Testenix 0.1 |
| --- | --- |
| `test` | One instance for one concrete test attempt. |
| `module` | Shared by normal tests from the module inside one worker. |
| `session` | Shared by normal tests assigned to one worker process. |

Session scope is currently worker-local, not a single process-global instance. Timed tests run in
dedicated processes and therefore receive isolated module/session fixtures.

## Custom fixture names

```python
@fixture(name="api")
def build_client() -> Client:
    return Client()


def test_health(api: Client) -> None:
    assert api.health() == "ok"
```

The custom name changes dependency lookup without changing the Python function.

---

# Document: Parallel execution

Canonical URL: https://polishdataengineer.github.io/testenix/guides/parallelism/
Source: docs/guides/parallelism.md

# Parallel execution

Testenix treats local parallelism as part of the execution model rather than an optional plugin.

## Choose the worker count

```console
$ testenix run --workers auto
$ testenix run --workers 4
$ testenix run --workers 1
```

`auto` resolves to the logical CPU count reported by Python. For CI and benchmark runs, an explicit
count makes resource use and results easier to reproduce.

## Scheduling

Normal tests from one module form an affinity unit and execute in the same persistent worker. This
preserves module-fixture reuse and avoids splitting hidden module state between processes.

When duration history exists, Testenix schedules longer units first. This longest-processing-time
strategy is deterministic and reduces the chance that one slow shard becomes the tail of the run.

## Process isolation

Workers are spawned processes. A worker streams each completed attempt to the coordinator before
starting the next test. If the worker later crashes:

- completed results remain authoritative;
- unfinished tests receive explicit crash or infrastructure outcomes;
- the crashed unit receives one framework recovery attempt;
- `CRASH -> PASS` is still finalized as `FLAKY`.

The coordinator never infers that a missing result passed.

## Hard timeouts

Any test with an explicit or global timeout runs in its own supervised process:

```python
from testenix import test


@test(timeout=2)
def test_external_tool() -> None:
    call_external_tool()
```

This boundary allows Testenix to terminate a blocking synchronous call as well as a stuck
coroutine. Descendant processes are terminated with the timed-out worker.

The trade-off is fixture reuse: a timed test cannot share module/session fixture instances with
neighbouring tests.

## Cancellation

The async library API is cancellation-aware:

```python
import asyncio

from testenix import TestenixConfig, run_async


async def main() -> None:
    result = await run_async(("tests",), TestenixConfig(workers=4))
    print(result.exit_code)


asyncio.run(main())
```

Cancelling `run_async` terminates active collection and execution process trees before the
coroutine returns control.

## Platform note

On Windows, scripts that call `run()` or `run_async()` directly must use the standard
multiprocessing guard:

```python
if __name__ == "__main__":
    main()
```

The `testenix` command-line program handles process startup itself.

---

# Document: Reports and history

Canonical URL: https://polishdataengineer.github.io/testenix/guides/reports/
Source: docs/guides/reports.md

# Reports and history

Every output adapter consumes the same immutable run result. A status cannot be green in the
console and red in JSON because both are derived from one model.

## Console

The console report is always enabled:

```console
$ testenix run tests
```

It prints test outcomes, failure details, and a final summary suitable for local development.

## JSON

```console
$ testenix run --json reports/testenix.json
```

The JSON report preserves the complete hierarchy:

```text
run
└── test
    └── attempt
        ├── setup
        ├── call
        └── teardown
```

Consumers can distinguish failed assertions, setup errors, teardown errors, timeouts, crashes,
expected failures, unexpected passes, flaky retries, and framework infrastructure errors.

## JUnit XML

```console
$ testenix run --junit reports/junit.xml
```

JUnit XML is designed for CI systems that already understand the common test-report format.
Testenix-specific identifiers and statuses are retained as properties where JUnit has no exact
equivalent.

## Duration history

By default, duration history is stored in `.testenix/history.sqlite3`. Later runs use the estimates
to schedule longer work earlier.

Choose a custom location:

```console
$ testenix run --history .cache/testenix.sqlite3
```

Disable all history writes:

```console
$ testenix run --no-history
```

History changes scheduling estimates, not test semantics. A test is never skipped or treated as
passing because of a historical record.

## Configure report paths

```toml
[tool.testenix]
json = "reports/testenix.json"
junit = "reports/junit.xml"
history = ".testenix/history.sqlite3"
```

CLI options override these paths for the current run.

## Exit codes

| Code | Meaning |
| ---: | --- |
| `0` | No gating failure. |
| `1` | Test failure, error, timeout, crash, unexpected pass, or flaky result. |
| `2` | Collection, command, configuration, or empty explicit selection error. |
| `3` | Internal runner, report, or history error. |
| `130` | Interrupted by the user. |

---

# Document: CLI reference

Canonical URL: https://polishdataengineer.github.io/testenix/reference/cli/
Source: docs/reference/cli.md

# Command-line reference

## Top-level command

```text
testenix [-h] [--version]
testenix [--config PYPROJECT] run [RUN_ARGS ...]
testenix pytest [PYTEST_ARGS ...]
testenix migrate FRAMEWORK PATH [PATH ...] [MIGRATION_ARGS ...]
```

| Option | Description |
| --- | --- |
| `-h`, `--help` | Show command help. |
| `--version` | Print the installed Testenix version. |
| `--config PATH` | Load `[tool.testenix]` for native `testenix run`. |

## `testenix run`

```text
testenix run [PATH ...]
             [--config PYPROJECT]
             [-w auto|N]
             [--retries N]
             [--timeout SECONDS]
             [-t TAG ...]
             [--json FILE]
             [--junit FILE]
             [--history FILE | --no-history]
```

| Argument | Default | Description |
| --- | --- | --- |
| `PATH ...` | configured `paths`, otherwise `tests` | Files or directories to discover. |
| `-w`, `--workers` | `auto` | Worker process count or logical CPU count. |
| `--retries` | `0` | Additional attempts after a gating outcome. |
| `--timeout` | none | Global hard deadline for every selected test. |
| `-t`, `--tag` | none | Required tag; repeat for AND selection. |
| `--json` | none | Write the lossless JSON run result. |
| `--junit` | none | Write a JUnit XML report. |
| `--history` | `.testenix/history.sqlite3` | Override the duration-history database. |
| `--no-history` | off | Disable reading and writing history. |

CLI options override `[tool.testenix]` values for the current run.

## `testenix pytest`

```text
testenix pytest [PYTEST_ARGS ...]
```

This compatibility command hands the current CLI process to pytest from the same interpreter and
forwards every argument without translation. It preserves pytest configuration, collection,
fixtures, parametrization, markers, classes, hooks, plugins, output, and normal exit status.

```console
$ testenix pytest -q tests
$ testenix pytest tests/test_api.py::test_health -k smoke --maxfail=1
$ testenix pytest -n auto tests
$ testenix pytest --junitxml=reports/pytest.xml tests
```

Pytest and its plugins must be installed in the same Python environment as Testenix. The optional
`testenix[pytest]` extra installs a supported pytest (`>=8.3,<10`) when needed. Testenix does not
consume a leading `--`: pytest receives it unchanged, including its normal end-of-options meaning.

`[tool.testenix]` and native options such as `--workers`, `--retries`, `--timeout`, `--tag`,
`--json`, `--junit`, and `--history` do not affect this command. Pass pytest or plugin options
instead. In particular, `testenix --config PATH pytest ...` is rejected; `pytest` must immediately
follow `testenix`. See [pytest compatibility](https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/) for the full
boundary.

## `testenix migrate`

```text
testenix migrate {auto,pytest,unittest} PATH [PATH ...]
                 [-o OUTPUT]
                 [-w auto|N]
                 [--validation-timeout SECONDS]
                 [--dry-run | --check]
                 [--report-json FILE|-]
```

| Argument | Default | Description |
| --- | --- | --- |
| `FRAMEWORK` | required | `pytest`, `unittest`, or `auto` for separate modules of both kinds. |
| `PATH ...` | required | Source files or directories. Sources are never modified. |
| `-o`, `--output` | `testenix_migrated` | New directory to publish; it must not exist. |
| `-w`, `--workers` | `auto` | Worker count for parallel candidate validation; an integer must be at least `2`. |
| `--validation-timeout` | `300` | Independent deadline for each source/native subprocess. |
| `--dry-run` | off | Static analysis and source-hash check only; run and publish nothing. |
| `--check` | off | Full differential validation, with no published output. |
| `--report-json` | none | New audit path inside the project and outside source/output suites, or `-` for clean JSON on standard output. |

Without `--dry-run` or `--check`, migration requires a green source baseline, an equal native
serial result, and an equal native parallel result, including every mapped test outcome. It then rechecks source hashes and atomically
renames a complete staging directory to the new output without replacement. A failure before that
rename leaves the output absent. A report-only failure after publication warns but leaves the
validated output and successful exit status intact. See [safe migration](https://polishdataengineer.github.io/testenix/guides/migration/) for supported constructs, unittest's
SHA-pinned wrapper model, rollback guarantees, and external-side-effect boundaries.

## Examples

```console
$ testenix run
$ testenix run tests/unit tests/integration --workers 4
$ testenix run --tag unit --tag fast
$ testenix run --retries 1 --timeout 10
$ testenix run --json reports/run.json --junit reports/junit.xml
$ testenix --config config/pyproject.toml run
$ testenix pytest -q tests
$ testenix migrate pytest tests --dry-run
$ testenix migrate auto tests --check --report-json reports/migration.json
$ testenix migrate unittest tests --output tests_testenix
```

## Exit codes

The native `testenix run` command uses these codes:

| Code | Meaning |
| ---: | --- |
| `0` | The run has no gating result. |
| `1` | A test failed, errored, timed out, crashed, became flaky, or unexpectedly passed. |
| `2` | Invalid CLI/configuration, collection error, or empty explicit tag selection. |
| `3` | Internal runner, reporter, or history failure. |
| `130` | User interruption. |

The `testenix migrate` command uses these codes:

| Code | Meaning |
| ---: | --- |
| `0` | Analysis, check, or publication succeeded; also retained after a report-only failure following publication. |
| `1` | Source/candidate execution failed, timed out, or differed. |
| `2` | Unsafe path, existing output, source drift, or invalid migration usage. |
| `3` | Internal process or report failure before publication. |
| `4` | An unsupported construct was found; nothing was published. |
| `130` | User interruption. |

`testenix pytest` hands the current CLI process to pytest, so pytest or plugin exit statuses are
returned unchanged. Standard pytest statuses are `0` success, `1` test failure, `2` interruption,
`3` internal error, `4` usage error, and `5` no tests collected. Before the handoff, Testenix
returns `2` when pytest is missing and `3` when pytest cannot take over the process.

---

# Document: Configuration reference

Canonical URL: https://polishdataengineer.github.io/testenix/reference/configuration/
Source: docs/reference/configuration.md

# Configuration reference

Project defaults live in `[tool.testenix]` in `pyproject.toml`.

This table configures only the native `testenix run` engine. The `testenix pytest` compatibility
command delegates configuration to pytest and therefore uses `pytest.ini`, `pyproject.toml`
`[tool.pytest.ini_options]`, or arguments passed after the subcommand.

```toml
[tool.testenix]
paths = ["tests"]
workers = "auto"
retries = 0
# timeout = 10.0
tags = []
# json = "reports/testenix.json"
# junit = "reports/junit.xml"
history = ".testenix/history.sqlite3"
```

Unknown options are rejected instead of being silently ignored.

## Options

### `paths`

- Type: string or list of strings
- Default: `["tests"]`

Files and directories used when no positional path is passed to `testenix run`.

### `workers`

- Type: positive integer or `"auto"`
- Default: `"auto"`

`auto` uses Python's logical CPU count. An explicit number is recommended for reproducible CI and
benchmark runs.

### `retries`

- Type: non-negative integer
- Default: `0`

Number of user-requested attempts after a gating outcome. Earlier failures remain part of the
result, so a later pass is finalized as `FLAKY`.

### `timeout`

- Type: positive finite number in seconds
- Default: none

Global hard timeout for selected tests. Timed tests run in isolated processes.

### `tags`

- Type: string or list of strings
- Default: empty

Every configured tag must be present on a test for it to be selected. A comma-separated string and
a TOML list are both accepted.

### `json`

- Type: filesystem path or `null`
- Default: none

Location of the lossless JSON report. The programmatic field is `json_path`.

### `junit`

- Type: filesystem path or `null`
- Default: none

Location of the JUnit XML report. The programmatic field is `junit_path`.

### `history`

- Type: filesystem path, `false`, or `null`
- Default: `.testenix/history.sqlite3`

Duration-history database. Set it to `false` to disable history in TOML:

```toml
[tool.testenix]
history = false
```

The programmatic field is `history_path` and accepts a `pathlib.Path` or `None`.

## Programmatic configuration

```python
from pathlib import Path

from testenix import TestenixConfig

config = TestenixConfig(
    paths=("tests/unit",),
    workers=4,
    retries=1,
    timeout=5.0,
    tags=("unit",),
    json_path=Path("reports/run.json"),
    history_path=None,
)
```

`TestenixConfig` is immutable. Use `with_overrides` to derive a validated copy.

---

# Document: Python API reference

Canonical URL: https://polishdataengineer.github.io/testenix/reference/api/
Source: docs/reference/api.md

# Python API reference

The objects below are exported from `testenix` and form the public pre-1.0 API. Pre-1.0 releases
may still make documented breaking changes.

## Authoring

```{eval-rst}
.. autofunction:: testenix.test

.. autofunction:: testenix.fixture

.. autofunction:: testenix.case

.. autofunction:: testenix.cases
```

### `skip`

```python
@skip(reason_or_function=None, /, *, reason=None, when=True)
```

Mark a test as skipped. It can be used as `@skip`, `@skip("reason")`, or with a conditional
`when=...`.

### `xfail`

```python
@xfail(reason_or_function=None, /, *, reason=None, when=True)
```

Mark a test as expected to fail. An unexpected pass becomes the gating `XPASS` status.

```{eval-rst}
.. autoclass:: testenix.CaseDefinition
   :members:
```

## Discovery and execution

```{eval-rst}
.. autofunction:: testenix.discover

.. autofunction:: testenix.run

.. autofunction:: testenix.run_async

.. autoclass:: testenix.TestenixConfig
   :members:
```

## Result contracts

```{eval-rst}
.. autoclass:: testenix.RunResult
   :members:

.. autoclass:: testenix.TestResult
   :members:

.. autoclass:: testenix.TestSpec
   :members:

.. autoclass:: testenix.CollectionResult
   :members:

.. autoclass:: testenix.Status
   :members:

.. autoclass:: testenix.Scope
   :members:
```

## Migration

The programmatic API follows the same copy-only transaction as `testenix migrate`. `project_root`
defaults to the current directory. Prefer `dry_run=True` before a validating or publishing call.

```python
from pathlib import Path

from testenix import MigrationOptions, migrate

report = migrate(
    MigrationOptions(
        framework="auto",
        sources=(Path("tests"),),
        output=Path("tests_testenix"),
        check_only=True,
    )
)
assert report.originals_modified is False
```

```{eval-rst}
.. autofunction:: testenix.migrate

.. autoclass:: testenix.MigrationOptions
   :members:

.. autoclass:: testenix.MigrationReport
   :members:

.. autoclass:: testenix.MigrationStatus
   :members:

.. autoclass:: testenix.ValidationSummary
   :members:
```

## Events

```{eval-rst}
.. autoclass:: testenix.Event
   :members:

.. autoclass:: testenix.EventSink
   :members:
```

---

# Document: Benchmark results

Canonical URL: https://polishdataengineer.github.io/testenix/benchmarks/results/
Source: docs/benchmarks/results.md

# Published benchmark results

These tables are generated from the raw JSON committed in `benchmarks/`. They are development
evidence for specific synthetic workloads, not a universal claim that Testenix is always faster
than pytest. `Testenix` in these results means the native `testenix run` engine. The
`testenix pytest` compatibility bridge delegates to pytest and is not represented here.

![Preliminary Testenix throughput ratios](https://polishdataengineer.github.io/testenix/_static/benchmark-speedup.svg)

## Median wall-clock time

Lower time is better. A speedup of `2.85×` means pytest's median
wall time was 2.85 times the Testenix median for that exact
scenario.

| Scenario | Testenix | pytest | pytest-xdist | vs pytest | vs xdist |
| --- | ---: | ---: | ---: | ---: | ---: |
| 10,000 no-op tests / 16 modules | 0.869 s | 2.477 s | 2.106 s | 2.85× | 2.42× |
| 10,000 uneven-duration tests / 16 modules | 1.345 s | 3.076 s | 2.138 s | 2.29× | 1.59× |
| 100,000 no-op tests / 16 modules | 8.038 s | 25.333 s | 21.300 s | 3.15× | 2.65× |

<div class="benchmark-caveat">
The 100,000-test result meets the project's local five-run, one-warmup minimum.
It remains a synthetic result from one machine, not a universal performance promise.
</div>

## Environment

- CPU: Apple M4 Pro (14 logical CPUs)
- Machine: `arm64`
- Platform: `macOS-26.5.1-arm64-arm-64bit`
- Python: `3.11.14`
- Measurement: complete subprocess wall-clock time, including discovery, execution, aggregation,
  and console rendering
- Correctness gate: every command had to exit successfully and report the expected test count



## Raw samples and variance

### 10,000 no-op tests / 16 modules

- Testenix range: 0.857 s–0.892 s
- Testenix standard deviation: 0.013 s
- Testenix raw samples: 0.869, 0.857, 0.863, 0.892, 0.871 seconds
- pytest range: 2.447 s–2.522 s
- pytest standard deviation: 0.027 s
- pytest raw samples: 2.522, 2.477, 2.471, 2.447, 2.479 seconds
- pytest-xdist range: 2.075 s–2.267 s
- pytest-xdist standard deviation: 0.081 s
- pytest-xdist raw samples: 2.170, 2.267, 2.077, 2.075, 2.106 seconds
- Measured rounds: 5; warmups: 1
- Workers: 4
- Recorded at: `2026-07-20T12:12:43.635798+00:00`
- Commit: `8f24f8a7bd72fa876988a8ce96364be97e35c2b6`
- Clean working tree at capture: yes
- [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/baseline.json)

### 10,000 uneven-duration tests / 16 modules

- Testenix range: 1.336 s–1.378 s
- Testenix standard deviation: 0.016 s
- Testenix raw samples: 1.336, 1.378, 1.342, 1.345, 1.356 seconds
- pytest range: 3.043 s–3.085 s
- pytest standard deviation: 0.016 s
- pytest raw samples: 3.070, 3.085, 3.043, 3.076, 3.077 seconds
- pytest-xdist range: 2.109 s–2.176 s
- pytest-xdist standard deviation: 0.025 s
- pytest-xdist raw samples: 2.146, 2.129, 2.109, 2.138, 2.176 seconds
- Measured rounds: 5; warmups: 1
- Workers: 4
- Recorded at: `2026-07-20T12:13:53.369799+00:00`
- Commit: `18d9bba6cb5c8e39c2d5b211ee4384ae8f824524`
- Clean working tree at capture: yes
- [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/baseline_uneven.json)

### 100,000 no-op tests / 16 modules

- Testenix range: 7.912 s–8.096 s
- Testenix standard deviation: 0.075 s
- Testenix raw samples: 7.912, 8.096, 8.086, 8.038, 7.997 seconds
- pytest range: 25.188 s–27.005 s
- pytest standard deviation: 0.772 s
- pytest raw samples: 27.005, 25.333, 25.246, 25.188, 25.380 seconds
- pytest-xdist range: 21.120 s–22.216 s
- pytest-xdist standard deviation: 0.486 s
- pytest-xdist raw samples: 21.239, 21.120, 22.216, 21.300, 21.949 seconds
- Measured rounds: 5; warmups: 1
- Workers: 4
- Recorded at: `2026-07-20T12:19:39.942492+00:00`
- Commit: `24b877c2f98420e91dcd2c8bcbc9417c7cf1ac96`
- Clean working tree at capture: yes
- [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/baseline_100k.json)

## Migrated-suite measurements

These separate measurements start with generated pytest or unittest sources, complete one safe
copy-and-validate migration, and then compare recurring source-suite runs with recurring native
Testenix runs. The migration transaction is a one-time cost shown separately; it is not included
in either execution median.

| Source runner | Workload | Tests / modules | Source median | Native median | Native vs source | Migration transaction |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| pytest (sequential) | no-op | 3,000 / 64 | 1.539 s | 0.521 s | 2.96× faster | 5.940 s |
| unittest outcome probe (sequential) | no-op | 3,000 / 64 | 0.161 s | 1.192 s | 7.40× slower | 6.742 s |
| unittest outcome probe (sequential) | 1 ms body | 3,000 / 64 | 4.066 s | 2.577 s | 1.58× faster | 17.251 s |

The native side used four workers. The source pytest and unittest outcome-probe baselines were
sequential, so these rows do not compare Testenix with pytest-xdist or another parallel unittest
runner. The unittest probe uses the standard-library loader and result semantics, then serializes
per-test outcomes for parity checking; its timing therefore includes that small audit overhead.
The no-op unittest wrappers are 7.40× slower than the probe because wrapper, loading, and
result-adaptation costs dominate an empty body. With 1 ms of synthetic work per unittest method,
parallel native execution is 1.58× faster in this 64-module layout. Module count and duration are
therefore material, and none of these synthetic rows predicts a specific real project.

### Raw migration samples and variance

### pytest / no-op

- Source command: `python -m pytest -q -p no:cacheprovider tests`
- Native command: `python -m testenix run testenix_migrated --workers 4 --no-history`
- Source median: 1.539 s
- Source range: 1.519 s–1.573 s;
  standard deviation: 0.020 s
- Source raw samples: 1.547, 1.535, 1.539, 1.573, 1.519 seconds
- Native Testenix median: 0.521 s
- Native Testenix range: 0.496 s–0.570 s;
  standard deviation: 0.030 s
- Native Testenix raw samples: 0.532, 0.498, 0.570, 0.496, 0.521 seconds
- Native workers: 4
- Measured rounds: 5; warmups: 1
- One-time copy, validation, and publication transaction: 5.940 s
- Integrity gates: 3,000 converted tests, matching source/native outcomes,
  original SHA-256 values unchanged
- Recorded at: `2026-07-20T16:38:49.510465+00:00`
- Source commit: [`3a51a901d268b061e9a87168300b41f3a2714a84`](https://github.com/polishdataengineer/testenix/commit/3a51a901d268b061e9a87168300b41f3a2714a84); worktree clean
- Lock SHA-256: `8ef0a9258aa5196bf2891f9da9f66c29bcf4e9bf297d178f3d4939cad36130cf`
- Versions: pytest=9.1.1, python=3.11.14, testenix=0.1.0, unittest=stdlib-3.11.14
- Environment: cpu_count=14, cpu_model=Apple M4 Pro, machine=arm64, platform=macOS-26.5.1-arm64-arm-64bit, python_implementation=CPython, python_version=3.11.14
- [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_pytest_3000.json)

### unittest / no-op

- Source command: `python -m testenix._unittest_probe --output <project>/.benchmark-unittest.json tests`
- Native command: `python -m testenix run testenix_migrated --workers 4 --no-history`
- Source median: 0.161 s
- Source range: 0.154 s–0.166 s;
  standard deviation: 0.004 s
- Source raw samples: 0.161, 0.160, 0.161, 0.154, 0.166 seconds
- Native Testenix median: 1.192 s
- Native Testenix range: 1.151 s–1.264 s;
  standard deviation: 0.051 s
- Native Testenix raw samples: 1.192, 1.171, 1.256, 1.151, 1.264 seconds
- Native workers: 4
- Measured rounds: 5; warmups: 1
- One-time copy, validation, and publication transaction: 6.742 s
- Integrity gates: 3,000 converted tests, matching source/native outcomes,
  original SHA-256 values unchanged
- Recorded at: `2026-07-20T16:39:05.030979+00:00`
- Source commit: [`3a51a901d268b061e9a87168300b41f3a2714a84`](https://github.com/polishdataengineer/testenix/commit/3a51a901d268b061e9a87168300b41f3a2714a84); worktree clean
- Lock SHA-256: `8ef0a9258aa5196bf2891f9da9f66c29bcf4e9bf297d178f3d4939cad36130cf`
- Versions: pytest=9.1.1, python=3.11.14, testenix=0.1.0, unittest=stdlib-3.11.14
- Environment: cpu_count=14, cpu_model=Apple M4 Pro, machine=arm64, platform=macOS-26.5.1-arm64-arm-64bit, python_implementation=CPython, python_version=3.11.14
- [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_unittest_3000.json)

### unittest / 1 ms body

- Source command: `python -m testenix._unittest_probe --output <project>/.benchmark-unittest.json tests`
- Native command: `python -m testenix run testenix_migrated --workers 4 --no-history`
- Source median: 4.066 s
- Source range: 4.023 s–4.075 s;
  standard deviation: 0.021 s
- Source raw samples: 4.075, 4.063, 4.066, 4.023, 4.067 seconds
- Native Testenix median: 2.577 s
- Native Testenix range: 2.454 s–2.644 s;
  standard deviation: 0.071 s
- Native Testenix raw samples: 2.577, 2.601, 2.571, 2.454, 2.644 seconds
- Native workers: 4
- Measured rounds: 5; warmups: 1
- One-time copy, validation, and publication transaction: 17.251 s
- Integrity gates: 3,000 converted tests, matching source/native outcomes,
  original SHA-256 values unchanged
- Recorded at: `2026-07-20T16:40:02.708652+00:00`
- Source commit: [`3a51a901d268b061e9a87168300b41f3a2714a84`](https://github.com/polishdataengineer/testenix/commit/3a51a901d268b061e9a87168300b41f3a2714a84); worktree clean
- Lock SHA-256: `8ef0a9258aa5196bf2891f9da9f66c29bcf4e9bf297d178f3d4939cad36130cf`
- Versions: pytest=9.1.1, python=3.11.14, testenix=0.1.0, unittest=stdlib-3.11.14
- Environment: cpu_count=14, cpu_model=Apple M4 Pro, machine=arm64, platform=macOS-26.5.1-arm64-arm-64bit, python_implementation=CPython, python_version=3.11.14
- [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_unittest_3000_delay_1ms.json)


## Interpretation

The checked-in results show that Testenix has low per-test overhead for large generated suites and
that its built-in process model is competitive with both sequential pytest and pytest-xdist in
those scenarios.

They do **not** yet answer how Testenix performs for import-heavy applications, complex fixture
graphs, assertion failures, real repositories, or different operating systems. Pytest also has a
far larger plugin and tooling ecosystem. Read the
[full performance analysis](https://polishdataengineer.github.io/testenix/performance-analysis/) for profiling details, memory notes,
implemented optimizations, and the Rust/PyO3 decision.

## Reproduce

Run the same harness from a locked development environment:

```console
$ uv sync --locked --dev --no-editable
$ uv run python benchmarks/run_benchmark.py --tests 10000 --workers 4 --repeats 5
$ uv run python benchmarks/run_benchmark.py --tests 10000 --workers 4 --repeats 5 --uneven
$ uv run python benchmarks/run_benchmark.py --tests 100000 --workers 4 --repeats 5
$ uv run python benchmarks/run_migration_benchmark.py --framework pytest --tests 3000 \
    --modules 64 --workers 4 --warmups 1 --repeats 5 \
    --output benchmarks/migration_baseline_pytest_3000.json
$ uv run python benchmarks/run_migration_benchmark.py --framework unittest --tests 3000 \
    --modules 64 --workers 4 --warmups 1 --repeats 5 \
    --output benchmarks/migration_baseline_unittest_3000.json
$ uv run python benchmarks/run_migration_benchmark.py --framework unittest --tests 3000 \
    --modules 64 --workers 4 --delay-ms 1 --warmups 1 --repeats 5 \
    --output benchmarks/migration_baseline_unittest_3000_delay_1ms.json
```

Review the [benchmarking contract](https://polishdataengineer.github.io/testenix/benchmarking/) before comparing or publishing new data.

---

# Document: Benchmarking contract

Canonical URL: https://polishdataengineer.github.io/testenix/benchmarking/
Source: docs/benchmarking.md

# Benchmarking contract

Testenix must be compared with both sequential pytest and pytest-xdist. Comparing only with sequential
pytest would overstate the value of built-in parallelism.

The benchmark suite will contain:

- 100, 1,000, and 10,000 no-op tests to measure discovery and framework overhead;
- tests with deliberately uneven durations to measure scheduler tail latency;
- module- and session-scoped fixture suites to measure fixture reuse;
- synchronous and asynchronous tests;
- passing, failing, skipped, xfailed, flaky, timed-out, and crashing tests;
- a worker-crash scenario that verifies every selected test reaches a terminal state;
- cold and warm history runs.

For every scenario record wall-clock duration, collection time, execution time, peak memory,
worker utilization, number of process starts, result completeness, and output size. Performance
claims require at least five runs per configuration and must publish the environment fingerprint.

The current v0.1 harness automates wall-clock samples and the environment fingerprint across 16
generated modules for a four-worker run. It also validates the completed test count, rotates runner
order between measured rounds, supports an explicit module count, and records throughput, mean, and
standard deviation. Pytest plugin autoloading and its cache provider are disabled, pytest-xdist is
loaded explicitly, and every tool runs from the generated suite directory so repository-level
pytest configuration does not affect the comparison. New output also records the commit, dirty
state, lockfile hash, timestamp, and installed framework versions. The remaining telemetry above is
the acceptance contract for the next harness iteration, not data claimed by the checked-in baseline
files.

Correctness wins over speed: a run with a missing, duplicated, or incorrectly finalized result is
invalid and excluded from performance comparisons.

## Pytest compatibility bridge

Measurements of `testenix pytest` must be reported separately from native `testenix run`
measurements. The compatibility command hands the current interpreter to pytest through a POSIX
process overlay or pytest's in-process console entry point on Windows; it does not execute tests
through the Testenix engine.

A compatibility-overhead comparison must use the same interpreter, working directory,
environment, pytest configuration, plugins, and arguments for both `python -m pytest ...` and
`testenix pytest ...`. Any difference measures adapter overhead only and must not be presented as a
Testenix execution speedup. Native comparisons continue to use `testenix run` and must validate
that both runners execute the same tests and produce equivalent outcomes.

Run the reproducible local harness with:

```bash
uv run python benchmarks/run_benchmark.py --tests 1000 --workers 4 --repeats 5
uv run python benchmarks/run_benchmark.py --tests 1000 --workers 4 --repeats 5 --uneven
uv run python benchmarks/run_benchmark.py --tests 10000 --modules 1000 --workers 4 --repeats 5
```

Maintainers can run the same comparison from GitHub's **Benchmarks** workflow and download its raw
JSON artifact. Shared GitHub runners are appropriate for reproducibility checks, not for silently
replacing the approved marketing baseline: their timing variance is outside this project's control.

The checked-in baseline files are development evidence, not universal performance claims. See
`docs/performance-analysis.md` for the current large-suite results, optimization profile, memory
notes, and native-code decision. Real project suites and cross-platform repetitions remain required
before publishing broad comparative claims.

An approved public baseline must be committed through a reviewed pull request. Do not remove slow
but valid samples as outliers; invalid commands remain evidence and must be explained. The current
checked-in 10,000- and 100,000-test files each contain five measured rounds, one warm-up, and clean
commit provenance. They remain single-machine synthetic evidence, so broader claims still require
the real-project and cross-platform scenarios above.

---

# Document: Performance analysis

Canonical URL: https://polishdataengineer.github.io/testenix/performance-analysis/
Source: docs/performance-analysis.md

# Performance analysis

## Executive summary

The optimized native v0.1 `testenix run` engine is faster than pytest and pytest-xdist in the checked-in synthetic
large-suite scenarios. The largest recorded comparison is 100,000 passing tests across 16 modules:
Testenix completed the suite in a median 8.038 seconds, pytest in 25.333 seconds, and pytest-xdist
in 21.300 seconds. Every measured command had to report the expected test count or the harness
rejected the sample. The baseline contains one warm-up and five counterbalanced measured rounds,
and Testenix's samples ranged from 7.912 to 8.096 seconds.

This is evidence for the tested workload and machine, not a universal claim about every Python
project. Import-heavy suites, fixture-heavy suites, slow tests, failure output, different operating
systems, and real repositories still need independent measurements.

These results do not apply to `testenix pytest`. The compatibility command delegates to pytest and
has pytest execution performance plus launcher and adapter overhead, which has not yet been
measured separately.

The new safe-migration baseline is deliberately mixed rather than uniformly positive. For 3,000
no-op pytest tests across 64 modules, the generated native suite was 2.96x faster than sequential
pytest. Empty unittest wrappers were 7.40x slower than the sequential stdlib-based outcome probe,
while the same layout with 1 ms of work in each method was 1.58x faster under four native workers. These are
synthetic, sequential-source comparisons; they do not establish an advantage over pytest-xdist,
parallel unittest runners, or a real repository.

## Environment and method

- Apple M4 Pro, 14 logical CPUs, 24 GiB RAM;
- macOS 26.5.1 arm64;
- CPython 3.11.14;
- four workers unless stated otherwise;
- generated test modules accepted by both Testenix and pytest;
- full process wall-clock time, including discovery, execution, aggregation, and console render;
- deterministic rotated execution order instead of always running Testenix last;
- test-count validation from each runner's final output;
- disabled pytest plugin autoloading and cache, with pytest-xdist loaded explicitly;
- generated-suite working directory, isolated from repository-level pytest configuration;
- `--no-history` for the primary runner-overhead comparison.

The harness records every sample, median, mean, standard deviation, environment fingerprint, and
median throughput. All checked-in comparison files use one warm-up and five measured repetitions.
They also record the clean source commit, lockfile hash, timestamp, CPU model, and installed
Testenix, pytest, and pytest-xdist versions.

## Results

| Scenario | pytest | pytest-xdist | Native Testenix | Native Testenix advantage |
|---|---:|---:|---:|---:|
| 10,000 no-op tests, 16 modules | 2.477 s | 2.106 s | 0.869 s | 2.85x vs pytest |
| 10,000 uneven tests, 16 modules | 3.076 s | 2.138 s | 1.345 s | 2.29x vs pytest |
| 100,000 no-op tests, 16 modules | 25.333 s | 21.300 s | 8.038 s | 3.15x vs pytest |

The 100,000-test median throughputs were 12,440 tests/s for Testenix, 3,947 tests/s for pytest, and
4,695 tests/s for pytest-xdist. The raw five-sample ranges and standard deviations are published in
the generated benchmark page and the checked-in JSON files.

In a separate exploratory 100,000-test profile, the coordinator's measured maximum resident set was
approximately 513 MiB after the final manifest/event optimization. Sequential pytest measured
approximately 520 MiB on the same generated suite. These older macOS `time` figures are not part of
the current baseline JSON and are process maxima, not aggregate memory across every xdist/Testenix
child process.

### Migrated-suite measurements

The migration harness generates a source suite, migrates and validates it without modifying any
source SHA-256, then measures the original source runner and the published native copy in
alternating rounds. All three checked-in scenarios contain 3,000 tests across 64 modules, use one
warm-up and five measured rounds, and run native Testenix with four workers on the M4 Pro/CPython
3.11 environment described above.

| Source runner | Test body | Source median | Native Testenix median | Native vs source | Migration transaction |
|---|---|---:|---:|---:|---:|
| sequential pytest | no-op | 1.539 s | 0.521 s | 2.96x faster | 5.940 s |
| sequential unittest outcome probe | no-op | 0.161 s | 1.192 s | 7.40x slower | 6.742 s |
| sequential unittest outcome probe | 1 ms sleep | 4.066 s | 2.577 s | 1.58x faster | 17.251 s |

The transaction duration is reported separately from recurring execution. It includes generation,
the source baseline, serial and parallel native candidates, parity checks, source rehashing, and
atomic publication. The 1 ms unittest transaction is longer because validation executes the
synthetic work repeatedly. Conversion is therefore an occasional safety cost, not a per-CI-run
speedup input.

The no-op unittest row exposes the adapter's fixed cost: each native test is a wrapper that loads
the unchanged source and translates the result of `TestCase.run()`. The sequential source probe
uses the stdlib loader and result semantics, then serializes per-test outcomes; it wins
when the test body does essentially nothing. With 1 ms per method, four-worker execution across 64
modules amortizes that cost and overtakes the sequential source runner. A suite concentrated in
one module would expose only one affinity unit, while too many workers can add process and import
overhead. Test duration, module distribution, imports, fixtures, I/O, failures, operating system,
and competing parallel runners must all be measured on the target project.

The generated [benchmark results](https://polishdataengineer.github.io/testenix/benchmarks/results/) publish every sample, range, standard
deviation, command, environment field, and raw JSON link. These three synthetic records are useful
for finding overhead boundaries; they are not evidence that converted suites are universally
faster.

### Worker-count sensitivity

An earlier exploratory run of 10,000 no-op tests across 16 modules produced these Testenix medians:

| Workers | Testenix median |
|---:|---:|
| 1 | 1.926 s |
| 4 | 1.483 s |
| 14 | 1.773 s |

Four workers were best for this short-test workload. This does not imply a universal four-worker
optimum: long CPU-bound tests can benefit from more processes. Adaptive worker selection should use
measured history rather than a fixed cap tuned to one machine.

### History and event-log cost

The original implementation coupled SQLite duration history to an event sink that opened, locked,
wrote, and closed its JSONL file for every event. A 10,000-test default-history run took 10.97
seconds. Keeping the descriptor open, avoiding internal fanout serialization, and storing one
self-contained attempt event instead of redundant post-hoc phase events reduced an equivalent run
to 1.98–2.30 seconds. Replay files still contain every test specification, complete attempt phases,
and final status.

## Profile and implemented optimizations

The first 10,000-test profile performed approximately 25.9 million Python calls. It spent large
fractions of coordinator time serializing the same 80,000 events more than once, reducing those
events again, resolving paths per test, and waiting for duplicated IPC payloads.

The optimized profile performed approximately 3.46 million calls. The main changes were:

1. Untimed synchronous tests reuse an executor thread instead of creating one OS thread per test.
   Timed tests retain a daemon-thread plus hard process deadline, so a stuck Python thread cannot
   block worker termination.
2. A successful worker streams each completed result for crash recovery and sends only a final ACK;
   it no longer sends the entire result tuple a second time.
3. Contract paths are computed once per module, source lines use `co_firstlineno`, and worker
   rediscovery builds an O(1) test-ID index instead of doing an O(m²) sequence of linear lookups.
4. The one-sink event path avoids JSON fanout. Duplicate event serialization is lazy and only runs
   when an event ID actually repeats.
5. Event IDs use the already unique `run_id:sequence` form instead of calling the OS random source
   tens of thousands of times.
6. Coordinator attempts are persisted as one complete replay event rather than a redundant
   started/three-phase/finished sequence emitted only after the attempt had already finished.
7. Unchanged selected specifications reuse discovered objects; selection events are omitted when
   every test is selected and no effective contract changed.
8. JSONL keeps one append descriptor for the run rather than performing open/close per event.

Correctness was retained throughout: the framework's full resilience suite passes after every
optimization, including worker crashes, timeout process-tree cleanup, collection crashes/hangs,
async task leaks, fixture teardown attribution, cancellation, retries, and event replay.

## Rust decision

Rust is not the next highest-value optimization. Python imports, Python test and fixture bodies,
and CPython object interaction remain Python work. PyO3 can release the interpreter only while
performing Rust-only work; it cannot make arbitrary Python callbacks execute in parallel under the
GIL.

The migration path does not change that decision. Parsing Python ASTs, hashing and copying files,
and publishing a new directory happen only when regenerating a suite, while most validation time
comes from executing the source and candidate Python tests. A Rust converter would not shorten a
1 ms Python test body or `unittest.TestCase.run()`. The empty-unittest result instead points to
profiling wrapper loading and result adaptation, then optimizing or caching those Python-level
boundaries without weakening SHA verification and fail-closed publication.

Direction-finding microbenchmarks on this machine showed:

| Candidate | Current signal | Rust/PyO3 result | Decision |
|---|---:|---:|---|
| LPT scheduling | ~2.1 ms for 1,000 synthetic units; real plan usually 16 units | bulk Rust loop ~17.5x faster | Absolute saving is below 2 ms; do not add native packaging for it |
| Event build + reduce | ~32 ms per 1,000 tests in a light microbenchmark | likely reducible in bulk | Revisit only if it exceeds 10% of final wall time |
| JSON event encoding | ~83 ms for ~8,000 detailed events | plausible 2–5x native gain | Relevant mainly to log-heavy mode; Python event compaction already recovered more |
| Pipe transport | 36.7 ms for 1,000 individual messages vs 12.2 ms as one batch | native framing could help | Batch/checkpoint in Python first |
| Sync invocation | 40–57 ms per 1,000 `to_thread` calls | Python body still crosses the GIL | Requires executor architecture, not Rust |
| Python list/record conversion | FFI conversion dominated the operation | PyO3 was slower than Python built-ins | Avoid fine-grained FFI calls |

If a later profile identifies a native-worthy data plane, the preferred design is an optional
PyO3 `abi3` extension with one bulk call per batch/run and a pure-Python fallback. Candidate scope:
framed IPC encoding/decoding or a bulk event reducer. Acceptance requires more than 10% end-to-end
improvement on Linux, macOS, and Windows with identical terminal results. A Rust sidecar or embedded
Python runtime is not justified: both retain Python interpreter startup/import costs while adding
another protocol and a much larger release matrix.

Relevant upstream constraints are documented in the
[PyO3 parallelism guide](https://pyo3.rs/main/parallelism),
[PyO3 performance guide](https://pyo3.rs/main/performance.html), and
[Maturin mixed-project guide](https://www.maturin.rs/project_layout.html).

## Next measurement gates

- collection, execution, IPC-byte, process-start, CPU, and aggregate-memory telemetry;
- 1,000/10,000/100,000 tests across 1, 16, 1,000, and 10,000 modules;
- synchronous and asynchronous fixtures, failures, captured output, timeouts, and retries;
- cold and warm SQLite history runs;
- real project suites and Linux/macOS/Windows CI runners;
- real migrated pytest and unittest suites, including sequential and established parallel source
  runners, multiple module layouts, and a break-even curve by median test duration;
- checkpoint-batched IPC followed by another profile;
- persistent workers that collect and execute without importing every module twice.

No universal “always faster than pytest” statement should be published until the real-project and
cross-platform gates pass. The supported claim today is narrower: Testenix is materially faster in the
measured native large passing-suite scenarios while retaining supervised isolation and complete
results.

---

# Document: Architecture

Canonical URL: https://polishdataengineer.github.io/testenix/architecture/
Source: docs/architecture.md

# Testenix architecture

Testenix is a native Python testing framework. Its core does not depend on pytest.
Compatibility adapters may translate foreign test frameworks into the same manifest and event
contracts, but they are not part of native execution.

The first pytest compatibility bridge deliberately does not translate pytest internals. On POSIX,
`testenix pytest` uses `os.execv` to replace itself with `sys.executable -m pytest`. On Windows it
calls pytest's public `console_main` entry point in the existing process because the platform's
`exec` family does not provide equivalent replacement semantics. Both paths keep pytest in the
foreground CLI process with the same working directory, environment, terminal, streams, signal
handling, and exit status. This preserves semantics without making the native core depend on
pytest:

```text
POSIX:   testenix pytest ==exec==> Python -m pytest -> collector/plugins/executor -> output/status
Windows: testenix pytest =========> pytest.console_main -> collector/plugins/executor -> output/status
```

The bridge is a CLI infrastructure adapter, not a native collection adapter. It does not emit
Testenix events or construct a `RunResult` in version 0.1.

The migration adapter is separate from that handoff. It statically converts a deliberately small
pytest subset or generates SHA-pinned wrappers around the standard unittest protocol. Its
application service owns a fail-closed transaction:

```text
immutable source snapshot
        |--- original runner in project shadow --------------------> baseline summary
        |--- generated copy -> native 1-worker shadow ------------> serial summary
        |--- generated copy -> native parallel shadow ------------> parallel summary
        +--- per-mapping/count/outcome/hash parity -> no-replace rename -> new output
```

Converters consume serializable `SourceFile` values and return generated artifacts, mappings, and
line-addressed diagnostics. They do not access the live filesystem. `migration_fs` is the only
publication adapter: it rejects link traversal and overlapping paths, creates private same-filesystem
staging, rehashes the source, and uses an operating-system atomic no-replace primitive. POSIX
staging creation, writes, publication, and cleanup remain anchored to captured directory identities;
platforms without safe recursive descriptor deletion retain a non-empty failed staging tree. A
pre-rename failure has no destination to roll back because the source was never a write target. A
post-rename durability or audit-report warning is reported as published rather than as a fictional
rollback.

Unittest wrappers deliberately keep original `TestCase.run()` as the semantic authority. They are
native scheduler units, but resolve the original through an exact wrapper-relative path and load
the class only after verifying the complete selected-Python-source SHA-256 manifest.
This avoids approximating setup, teardown, cleanup, assertion, mocking, async, skip, and expected
failure behavior.

## Product contract

Testenix aims to be typed, async-native, parallel-first, deterministic, and lossless when reporting
test outcomes. A retry never overwrites an earlier attempt, infrastructure failures are distinct
from test failures, and setup/call/teardown are preserved as separate phases.

```text
Authoring API -> supervised collection -> inert manifest -> affinity scheduler -> process workers
                                                                  |          -> streamed attempts
                                                                  +----------> append-only events
                                                                              -> reducer
                                                                              -> reports/history
```

## Dependency rules

- `contracts` contains serializable domain values and imports no infrastructure.
- `api`, `discovery`, `fixtures`, and `executor` form the native engine.
- `events`, `aggregate`, and `scheduler` remain engine-independent.
- `runner` is the application service connecting the native engine with execution policy.
- reporters and storage consume completed domain results or versioned events.
- optional compatibility adapters stay at the CLI boundary; the native core never imports pytest.
- migration analyzers depend on serializable migration contracts, while shadow execution and
  atomic publication remain application/infrastructure concerns.

## Version 0.1 scope

- explicit `@test` and `@fixture` authoring API, plus conventional `test_*` discovery;
- sync functions, coroutines, generators, and async-generator fixture teardown;
- explicit cases, tags, skip, expected failure, and per-test timeout metadata;
- fixture scopes: test, module, session, with broader scopes currently bounded by a worker shard;
- sequential and local process execution;
- deterministic scheduling based on historical durations;
- append-only JSONL events and a pure reducer;
- console, JSON, and JUnit output plus local SQLite duration history;
- retries represented as immutable attempts and finalized as `FLAKY` when appropriate;
- an optional platform-aware pytest handoff for unchanged legacy suites.
- conservative pytest/unittest migration with static diagnostics, differential validation, source
  fingerprints, and create-only publication.

Remote workers, distributed storage, result caching, automatic quarantine, and a stable third-party
plugin SDK are deliberately outside version 0.1.

## Fixture scopes and process isolation

The scheduler treats every normal test module as one affinity unit and never splits that unit
between parallel shared workers. Multiple modules assigned to one shard execute in one persistent
process and fixture runtime. A test with an explicit timeout (including a global timeout applied at
selection) is instead a single-test isolation unit with a hard process deadline.

Scope therefore has the following concrete meaning in version 0.1:

| Scope | Lifetime |
| --- | --- |
| `test` | One instance for one concrete test attempt. |
| `module` | One instance for all normal attempts from that module in a shared worker; a timed test has an isolated instance. |
| `session` | One instance for all normal tests assigned to a shared worker process; one per isolated timed process. |

Session fixtures are intentionally not run-global. Suites that require a true global singleton
must keep that resource outside the fixture runtime until a coordinator-owned resource protocol is
implemented. Setting one worker gives one shared session for normal tests, but timed tests remain
isolated by design.

## Worker protocol and recovery

Workers receive only primitive rediscovery locators (path, function/case identity, and effective
timeout), so arbitrary case values never cross the spawn boundary. A worker streams each completed
`AttemptResult` before starting the next test and later republishes an owner if session teardown
changes its result. The final batch envelope is still sent for normal completion.

The supervisor drains the pipe while the child runs, avoiding pipe-buffer deadlocks for large
results. If a later test crashes the process, already streamed attempts remain authoritative and
only unfinished tests receive infrastructure/crash outcomes. A lost final ACK after all provisional
results is a teardown failure, never a green run. A process crash gets one automatic recovery
attempt without consuming the user's retry budget, but `CRASH -> PASS` remains `FLAKY`; only a
framework-owned `INFRA_ERROR -> PASS` recovery becomes plain `PASS`.

Collection itself uses the same spawn-based supervisor and returns only a JSON-safe manifest. A
top-level import crash or deadline becomes a `CollectionIssue`, so user import code cannot exit or
indefinitely block the coordinator. Timed execution units send a ready handshake after rediscovery;
the test deadline therefore does not include interpreter startup or module import.

Workers normally create a separate POSIX process session (or use recursive tree termination on
Windows). During migration validation they remain in the validator's process group so an outer
validation deadline can terminate native workers too. Timeout and cancellation terminate ordinary
descendants started by a test as well as the direct worker; this is process supervision, not an OS
sandbox against a test that deliberately detaches itself.

---

# Document: Roadmap

Canonical URL: https://polishdataengineer.github.io/testenix/roadmap/
Source: docs/roadmap.md

# Roadmap

## 0.1 — native vertical slice

- Native function tests and explicit test descriptions.
- Typed fixture graph with test, module, and session scopes.
- Sync, coroutine, generator, and async-generator execution.
- Explicit cases, tags, skip, expected failure, and timeouts.
- Local process workers with deterministic LPT scheduling.
- Immutable attempts and lossless setup/call/teardown aggregation.
- Console, JSON, JUnit XML, JSONL events, and SQLite duration history.
- Optional `testenix pytest` compatibility bridge preserving the real pytest engine and plugins.
- Safe native migration for a conservative pytest subset and direct unittest TestCase classes,
  with source fingerprints, disposable validation copies, serial/parallel parity, and atomic
  create-only publication.

## 0.2 — fast feedback

- Dynamic micro-shards and work stealing.
- `--last-failed`, watch mode, and failure fingerprints.
- Test-impact analysis in shadow mode with an explanation for every selection decision.
- Stable assertion-diff protocol and improved plain-assert diagnostics.

## 0.3 — adoption

- Pytest hook adapter translating collection and outcomes into Testenix events and `RunResult`.
- Expand migration beyond the v0.1 static subset only when new transformations have differential
  semantics tests on real projects.
- Versioned reporter and selector plugin interfaces.
- IDE protocol and machine-readable collection manifest.

## Later

- Remote workers with leases and heartbeats.
- Resource-capacity scheduling and explicit test affinity.
- Opt-in caching for tests that declare all external inputs.
- Flakiness history, quarantine ownership, and expiration policy.

Result caching and impact selection will not become build-gating defaults until their false-negative
rate is measured continuously against full-suite runs.

---

# Document: Changelog

Canonical URL: https://github.com/polishdataengineer/testenix/blob/main/CHANGELOG.md
Source: CHANGELOG.md

# Changelog

All notable changes will be documented in this file. The format follows Keep a Changelog and the
project intends to use Semantic Versioning once its public API reaches stability.

## [Unreleased]

### Added

- `testenix pytest [PYTEST_ARGS ...]` compatibility bridge for unchanged pytest suites, preserving
  the real pytest collector, fixtures, parametrization, markers, plugins, output, and exit status.
- Optional `testenix[pytest]` installation extra and a documented native-versus-compatibility
  capability matrix.
- `testenix migrate {auto,pytest,unittest}` with dry-run, full check, JSON audit reporting, source
  SHA-256 snapshots, disposable project shadows, serial/parallel differential validation, and
  atomic create-only publication to a new directory.
- Conservative pytest transformations for static module functions, cases, simple fixtures,
  adjacent `conftest.py`, skips, and selection markers, with stable diagnostics for unsupported
  semantics.
- Native unittest wrapper generation that retains the original `TestCase.run()` lifecycle and
  assertions while resolving sources independently of `cwd` and pinning every selected Python
  module through a SHA-256 manifest.
- A reproducible migrated-suite benchmark harness for 3,000+ pytest or unittest tests, including
  count/hash gates, counterbalanced rounds, environment provenance, and raw JSON output.
- Per-test source-to-target outcome parity, configured parallel validation with an explicit
  one-affinity-unit warning, bounded descendant cleanup, disjoint create-only audit reports, and
  descriptor-anchored POSIX staging creation, writes, publication, and cleanup.
- Truthful post-commit durability/report warnings, package-aware unittest outcome mapping,
  Testenix validation-worker containment, and conservative blocking of pytest session fixtures and
  unittest class-cleanup hooks whose lifecycle cannot be preserved.

## [0.1.0] - 2026-07-20

### Added

- Native authoring API, fixture graph, cases, and sync/async execution.
- Versioned event stream and lossless run/test/attempt/phase result model.
- Deterministic local scheduling, process-worker supervision, and retries.
- Console, JSON, JUnit XML, and SQLite history adapters.
- `testenix` command-line interface and `pyproject.toml` configuration.
- Supervised collection, worker-ready handshakes, process-tree cleanup, and cancellable async runs.
- Module affinity, streamed partial-result recovery, and stable rediscovery locators for arbitrary
  case values.
- Strict mypy/pyright validation and adversarial tests for crashes, hangs, retries, and teardown
  ownership.
- Counterbalanced, count-validating 1k/10k/100k performance harness with configurable module count
  and scalable uneven workloads.
- A searchable Sphinx/Furo documentation site for GitHub Pages, generated Python API reference,
  and one-click page/full-document copying for LLM context.
- Deterministic `llms.txt` and `llms-full.txt` references plus generated benchmark tables and SVG.
- Manual GitHub benchmark workflow with downloadable raw JSON results.

### Changed

- Reused sync executor threads, removed duplicate final IPC payloads, indexed worker rediscovery,
  and eliminated per-test path/source inspection.
- Compacted coordinator events, made event IDs deterministic, retained one JSONL descriptor per
  run, and removed unchanged manifest/selection copies.
- Reduced the recorded 100k-test median to 8.038 seconds versus 25.333 seconds for pytest on the
  documented M4 Pro development baseline; comparative claims remain workload-specific.
- Isolated benchmark runs from repository pytest configuration, disabled pytest cache/plugin
  autoloading, and added commit, lockfile, version, and dirty-state provenance to new results.
- Hardened PyPI releases by requiring the release tag to reference a commit contained in `main`.

---

# Document: Security policy

Canonical URL: https://github.com/polishdataengineer/testenix/blob/main/SECURITY.md
Source: SECURITY.md

# Security policy

## Supported versions

Testenix is currently pre-1.0. Security fixes are applied to the latest released version.

## Reporting a vulnerability

Please do not open a public issue for a suspected vulnerability. Use GitHub's private
vulnerability reporting for this repository instead. Include the affected version, a minimal
reproduction, impact, and any suggested mitigation. We will acknowledge a complete report within
seven days and coordinate disclosure after a fix is available.

---

# Expanded public API snapshot

This section is generated from `testenix.__all__` so an LLM receives concrete symbols, signatures, enum values, dataclass fields, and docstrings rather than unresolved documentation directives.

## `testenix.CaseDefinition`

```text
CaseDefinition(parameters: 'Mapping[str, Any]', id: 'str | None' = None) -> None
```

Fields:

- `parameters`: `Mapping[str, Any]`
- `id`: `str | None`

One concrete parameter mapping attached to a test function.

## `testenix.CollectionResult`

```text
CollectionResult(items: 'tuple[CollectedTest, ...]', registry: 'FixtureRegistry', issues: 'tuple[CollectionIssue, ...]' = ()) -> None
```

Fields:

- `items`: `tuple[CollectedTest, ...]`
- `registry`: `FixtureRegistry`
- `issues`: `tuple[CollectionIssue, ...]`

Complete collection output, including non-fatal authoring issues.

## `testenix.Event`

```text
Event(event_id: 'str', run_id: 'str', event_type: 'EventType', timestamp: 'float', sequence: 'int', schema_version: 'int' = 1, test_id: 'str | None' = None, attempt: 'int | None' = None, worker_id: 'str | None' = None, payload: 'dict[str, Any]' = <factory>) -> None
```

Fields:

- `event_id`: `str`
- `run_id`: `str`
- `event_type`: `EventType`
- `timestamp`: `float`
- `sequence`: `int`
- `schema_version`: `int`
- `test_id`: `str | None`
- `attempt`: `int | None`
- `worker_id`: `str | None`
- `payload`: `dict[str, Any]`

Append-only fact emitted while a run is executing.

## `testenix.EventSink`

```text
EventSink(*args, **kwargs)
```

Minimal sink contract used by runners, workers and adapters.

## `testenix.MigrationOptions`

```text
MigrationOptions(framework: 'Framework', sources: 'tuple[Path, ...]', output: 'Path' = PosixPath('testenix_migrated'), workers: 'WorkerCount' = 'auto', validation_timeout: 'float' = 300.0, dry_run: 'bool' = False, check_only: 'bool' = False, project_root: 'Path | None' = None) -> None
```

Fields:

- `framework`: `Framework`
- `sources`: `tuple[Path, ...]`
- `output`: `Path`
- `workers`: `WorkerCount`
- `validation_timeout`: `float`
- `dry_run`: `bool`
- `check_only`: `bool`
- `project_root`: `Path | None`

Validated input for :func:`migrate`.

## `testenix.MigrationReport`

```text
MigrationReport(status: 'MigrationStatus', mode: 'str', framework: 'str', project_root: 'str', sources: 'tuple[str, ...]', output: 'str', source_hashes: 'Mapping[str, str]', generated_files: 'tuple[str, ...]', mappings: 'tuple[TestMapping, ...]', diagnostics: 'tuple[MigrationDiagnostic, ...]', baseline: 'ValidationSummary | None', native_serial: 'ValidationSummary | None', native_parallel: 'ValidationSummary | None', originals_modified: 'bool', published: 'bool', message: 'str', started_at: 'float', finished_at: 'float') -> None
```

Fields:

- `status`: `MigrationStatus`
- `mode`: `str`
- `framework`: `str`
- `project_root`: `str`
- `sources`: `tuple[str, ...]`
- `output`: `str`
- `source_hashes`: `Mapping[str, str]`
- `generated_files`: `tuple[str, ...]`
- `mappings`: `tuple[TestMapping, ...]`
- `diagnostics`: `tuple[MigrationDiagnostic, ...]`
- `baseline`: `ValidationSummary | None`
- `native_serial`: `ValidationSummary | None`
- `native_parallel`: `ValidationSummary | None`
- `originals_modified`: `bool`
- `published`: `bool`
- `message`: `str`
- `started_at`: `float`
- `finished_at`: `float`

Complete audit record for a migration attempt.

## `testenix.MigrationStatus`

```text
MigrationStatus(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)
```

Values: ANALYZED='analyzed', VALIDATED='validated', PUBLISHED='published', UNSUPPORTED='unsupported', VALIDATION_FAILED='validation_failed', SAFETY_ERROR='safety_error'

Terminal state of one migration transaction.

## `testenix.TestenixConfig`

```text
TestenixConfig(paths: 'tuple[str, ...]' = ('tests',), workers: "int | Literal['auto']" = 'auto', retries: 'int' = 0, timeout: 'float | None' = None, tags: 'tuple[str, ...]' = (), json_path: 'Path | None' = None, junit_path: 'Path | None' = None, history_path: 'Path | None' = PosixPath('.testenix/history.sqlite3')) -> None
```

Fields:

- `paths`: `tuple[str, ...]`
- `workers`: `int | Literal['auto']`
- `retries`: `int`
- `timeout`: `float | None`
- `tags`: `tuple[str, ...]`
- `json_path`: `Path | None`
- `junit_path`: `Path | None`
- `history_path`: `Path | None`

Validated execution and reporting settings.

Paths are kept relative to the process working directory.  This mirrors
normal command-line path handling and makes a config object safe to pass to
a worker process without retaining hidden project state.

## `testenix.RunResult`

```text
RunResult(run_id: 'str', tests: 'tuple[TestResult, ...]', collection_issues: 'tuple[CollectionIssue, ...]', started_at: 'float', finished_at: 'float') -> None
```

Fields:

- `run_id`: `str`
- `tests`: `tuple[TestResult, ...]`
- `collection_issues`: `tuple[CollectionIssue, ...]`
- `started_at`: `float`
- `finished_at`: `float`

RunResult(run_id: 'str', tests: 'tuple[TestResult, ...]', collection_issues: 'tuple[CollectionIssue, ...]', started_at: 'float', finished_at: 'float')

## `testenix.Scope`

```text
Scope(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)
```

Values: TEST='test', MODULE='module', SESSION='session'

Lifetime of a fixture instance.

## `testenix.Status`

```text
Status(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)
```

Values: PASS='pass', FAIL='fail', ERROR_SETUP='error_setup', ERROR_TEARDOWN='error_teardown', SKIP='skip', XFAIL='xfail', XPASS='xpass', TIMEOUT='timeout', CRASH='crash', INFRA_ERROR='infra_error', CANCELLED='cancelled', NOT_RUN='not_run', FLAKY='flaky', CACHED_PASS='cached_pass'

A lossless outcome vocabulary for phases, attempts, and tests.

## `testenix.TestResult`

```text
TestResult(test: 'TestSpec', status: 'Status', attempts: 'tuple[AttemptResult, ...]', duration: 'float') -> None
```

Fields:

- `test`: `TestSpec`
- `status`: `Status`
- `attempts`: `tuple[AttemptResult, ...]`
- `duration`: `float`

TestResult(test: 'TestSpec', status: 'Status', attempts: 'tuple[AttemptResult, ...]', duration: 'float')

## `testenix.TestSpec`

```text
TestSpec(id: 'str', path: 'str', module_name: 'str', function_name: 'str', display_name: 'str', parameters: 'dict[str, Any]' = <factory>, case_id: 'str | None' = None, tags: 'frozenset[str]' = <factory>, skip_reason: 'str | None' = None, xfail_reason: 'str | None' = None, timeout: 'float | None' = None, source_line: 'int | None' = None) -> None
```

Fields:

- `id`: `str`
- `path`: `str`
- `module_name`: `str`
- `function_name`: `str`
- `display_name`: `str`
- `parameters`: `dict[str, Any]`
- `case_id`: `str | None`
- `tags`: `frozenset[str]`
- `skip_reason`: `str | None`
- `xfail_reason`: `str | None`
- `timeout`: `float | None`
- `source_line`: `int | None`

Serializable description of one concrete test case.

## `testenix.ValidationSummary`

```text
ValidationSummary(runner: 'str', tests: 'int', passed: 'int', failed: 'int', errors: 'int', skipped: 'int', xfailed: 'int', xpassed: 'int', exit_code: 'int', duration: 'float', timed_out: 'bool' = False, detail: 'str | None' = None, outcomes: 'Mapping[str, str]' = <factory>) -> None
```

Fields:

- `runner`: `str`
- `tests`: `int`
- `passed`: `int`
- `failed`: `int`
- `errors`: `int`
- `skipped`: `int`
- `xfailed`: `int`
- `xpassed`: `int`
- `exit_code`: `int`
- `duration`: `float`
- `timed_out`: `bool`
- `detail`: `str | None`
- `outcomes`: `Mapping[str, str]`

Framework-neutral outcome counts from one isolated validation run.

## `testenix.case`

```text
case(values: 'Mapping[str, Any] | str | None' = None, /, *, id: 'str | None' = None, **parameters: 'Any') -> 'CaseDefinition'
```

Attach one concrete parameter case.

Examples::

    @case(user="alice", expected=True)
    @case("anonymous", user=None, expected=False)
    @case({"id": 42}, id="record-42")

## `testenix.cases`

```text
cases(*definitions: 'CaseDefinition | Mapping[str, Any]', ids: 'Sequence[str] | None' = None, **dimensions: 'Iterable[Any]') -> 'Callable[[F], F]'
```

Attach several cases, either explicitly or as a Cartesian product.

``@cases({"x": 1}, {"x": 2})`` declares explicit mappings, while
``@cases(role=["admin", "editor"], active=[True, False])`` creates the
Cartesian product of the supplied dimensions.

## `testenix.discover`

```text
discover(paths: 'str | Path | Iterable[str | Path]' = '.') -> 'CollectionResult'
```

Discover native tests and fixtures below one or more paths.

Directories are searched recursively for ``test_*.py``.  An explicitly
supplied Python file may have any name, which is useful for focused runs.
Inside a module, ``test_*`` functions and functions decorated with
``@test`` (or parameterized with ``@case(s)``) are collected.

## `testenix.fixture`

```text
fixture(function: 'Callable[..., Any] | None' = None, /, *, scope: 'Scope | str' = <Scope.TEST: 'test'>, name: 'str | None' = None) -> 'Callable[..., Any]'
```

Declare a dependency provider with test, module or session lifetime.

## `testenix.migrate`

```text
migrate(options: 'MigrationOptions') -> 'MigrationReport'
```

Analyze, validate, and optionally publish a native Testenix copy.

No branch of this function writes to a source path.  A destination is only
published by a no-overwrite rename after the source fingerprints have been
checked for concurrent changes.

## `testenix.run`

```text
run(paths: 'Sequence[str] | str | None' = None, config: 'TestenixConfig | None' = None, *, event_sink: 'EventSink | None' = None) -> 'RunResult'
```

Discover and execute a native Testenix suite.

The coordinator is the only component that decides final outcomes. Workers
return immutable attempts, which are emitted to a versioned event stream and
reduced after execution.

## `testenix.run_async`

```text
run_async(paths: 'Sequence[str] | str | None' = None, config: 'TestenixConfig | None' = None, *, event_sink: 'EventSink | None' = None) -> 'RunResult'
```

Cancellable embedding facade around the process-oriented coordinator.

## `testenix.skip`

```text
skip(reason_or_function: 'str | F | None' = None, /, *, reason: 'str | None' = None, when: 'bool' = True) -> 'F | Callable[[F], F]'
```

Mark a test as skipped, optionally only when a condition is true.

## `testenix.test`

```text
test(function_or_description: 'Callable[..., Any] | str | None' = None, /, *, description: 'str | None' = None, tags: 'Iterable[str] | str' = (), timeout: 'float | None' = None) -> 'Callable[..., Any]'
```

Mark a function as a native Testenix test.

Supported forms are ``@test``, ``@test()``, ``@test("description")`` and
``@test(description="description", tags={...}, timeout=...)``.

## `testenix.xfail`

```text
xfail(reason_or_function: 'str | F | None' = None, /, *, reason: 'str | None' = None, when: 'bool' = True) -> 'F | Callable[[F], F]'
```

Mark a test as expected to fail, optionally only when a condition is true.
