Metadata-Version: 2.5
Name: testudos
Version: 0.43.0
Summary: A testing harness for Python packages using uv's isolated environments
Project-URL: Homepage, https://github.com/martinristovski/testudos
Project-URL: Documentation, https://github.com/martinristovski/testudos/blob/main/README.md
Project-URL: Repository, https://github.com/martinristovski/testudos
Project-URL: Issues, https://github.com/martinristovski/testudos/issues
Author: Martin Ristovski
License: MIT
License-File: LICENSE
Keywords: multi-version,python,testing,uv
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: packaging>=25.0
Requires-Dist: rich>=14.0.0
Requires-Dist: typer>=0.21.0
Provides-Extra: coverage
Requires-Dist: coverage>=7.13.0; extra == 'coverage'
Provides-Extra: dev
Requires-Dist: coverage>=7.13.0; extra == 'dev'
Requires-Dist: mypy>=1.19.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=1.3.0; extra == 'dev'
Requires-Dist: pytest-cov>=7.0.0; extra == 'dev'
Requires-Dist: pytest-xdist>=3.8.0; extra == 'dev'
Requires-Dist: pytest>=9.0.0; extra == 'dev'
Requires-Dist: ruff>=0.14.0; extra == 'dev'
Description-Content-Type: text/markdown

# testudos

A testing harness for Python packages using [uv](https://docs.astral.sh/uv/)'s isolated environments.

## Overview

Testudos runs your test suite across every Python version that is both still
supported upstream and compatible with your package. It works out which
versions those are by intersecting two sources:

1. **Currently supported Python versions** — from the [endoflife.date](https://endoflife.date/python) API (v1)
2. **Your package's compatibility** — the `requires-python` field in your `pyproject.toml`

Cycles that endoflife.date lists but that have no stable release yet (a
Python still in alpha, beta or rc) are left out unless you ask for them with
`--include-prerelease`.

## Quick start

```bash
# Install
uv tool install testudos

# See which versions would be tested, and why
testudos versions

# Run the suite on all of them
testudos run

# Run them concurrently
testudos run --parallel

# Show the exact commands without running anything
testudos run --dry-run
```

## Installation

```bash
uv tool install testudos    # recommended
pip install testudos
```

Coverage needs coverage.py on your machine as well as in the test
environments: `testudos run --coverage` combines and reports the data with it,
and so do `testudos coverage combine` and `testudos coverage report`. Install
the extra:

```bash
uv tool install 'testudos[coverage]'
pip install 'testudos[coverage]'
```

testudos runs the coverage installed alongside it, so the extra works under
`uv tool install`, which keeps it out of your `PATH`. Without the extra, a
`coverage` on `PATH` is used instead. Each isolated environment gets its own
copy too, pinned to the same version so the data files stay readable.

### Requirements

- Python 3.11 or newer
- [uv](https://docs.astral.sh/uv/)
- Linux or macOS. On other platforms testudos runs, but a timeout or an
  interrupt stops `uv` and can leave the test process it started running, and
  testudos warns about this when a run starts.

## How it works

For each Python version, testudos runs your tests in an ephemeral environment
created by `uv run --isolated`:

```bash
uv run --isolated --python=3.12 --with pytest --directory /path/to/project pytest
```

- `--isolated` builds a throwaway environment, so nothing leaks between versions
- `--python=X.Y` selects the interpreter, downloading it if needed
- `--with pytest` puts the test runner *inside* that environment, where it can
  see your project's dependencies

`--dry-run` prints exactly these commands, built by the same code that runs
them, so what you see is what would execute.

### Test dependencies

`uv run --isolated` installs your project, its runtime dependencies, and its
`dev` dependency group (`[dependency-groups] dev`), which uv includes by
default. Other groups and extras are not installed unless you name them. If
your test dependencies live in a group or an extra, name that: they are then
installed at the versions in `uv.lock`.

```bash
testudos run --group test           # [dependency-groups] test
testudos run --extra test           # [project.optional-dependencies] test
```

```toml
[tool.testudos]
test-groups = ["test"]
test-extras = ["test"]
```

(`[tool.uv] default-groups = ["dev", "test"]` in your project has the same
effect for every `uv run`, testudos' included.)

For anything not declared in the project, name the package itself:

```bash
testudos run --with pytest-asyncio --with pytest-mock
```

```toml
[tool.testudos]
test-dependencies = ["pytest-asyncio", "pytest-cov>=7"]
```

These are installed outside the lockfile, so a requirement without `==`
resolves to the newest release on every run: a new plugin release can change a
run with no change to your project, and a freshly published release reaches
every pipeline at once. `--dry-run` lists them as "Unpinned --with". Prefer a
group or an extra; otherwise pin them, or pass `--exclude-newer 2026-10-01`
(config `exclude-newer`) so resolution ignores anything released after that
date. The same applies to a project with no `uv.lock` -- a Poetry or setuptools
project, say -- whose whole dependency tree is resolved fresh each run, since uv
does not read other tools' lock files.

### The lockfile

With a `uv.lock`, each environment is installed from it. If `pyproject.toml`
has changed since the lock was written, uv re-resolves in memory, says nothing
and leaves `uv.lock` alone, so the run tests something other than what is
committed. In CI you usually want that to fail:

```bash
testudos run --locked     # fail when uv.lock is out of date
testudos run --frozen     # use uv.lock as it is, without checking it
```

`locked = true` / `frozen = true` in `[tool.testudos]` do the same, and
`--no-locked` / `--no-frozen` override them. `UV_LOCKED=1` in the environment
works too.

## Usage

### Choosing versions

```bash
testudos run                                # auto-detect
testudos run --python 3.11 --python 3.12    # explicit (-p)
testudos run --python 3.14.0rc1             # pre-releases are allowed explicitly
testudos run --include-prerelease           # include unreleased cycles when auto-detecting
testudos run --offline                      # skip the API, use cached or bundled data
testudos run --no-offline                   # refuse a configured offline = true
testudos run ./path/to/project              # a project other than the cwd
testudos --debug run                        # show the traceback behind a failure
```

`testudos versions` shows the same resolution as a table, marking each cycle
stable or pre-release and naming any pre-release it left out. For scripts and
CI it also prints JSON or a plain list:

```bash
testudos versions --format json     # {"versions": [...], "version_source": "api", "supported": [...]}
testudos versions --format plain    # 3.11 3.12 3.13 3.14
uv python install $(testudos versions --format plain)
```

#### Deciding the matrix centrally

endoflife.date moves on its own schedule, so the same commit can test a
different matrix the day a Python ships or reaches end of life. An organisation
that sets its own policy can name its supported cycles once, in a shared CI
template or the runner environment:

```bash
export TESTUDOS_PYTHON_VERSIONS="3.11 3.12 3.13"   # stands in for endoflife.date
```

The list replaces endoflife.date (no network call is made) and is still
intersected with each project's `requires-python`. A project's own
`python-versions`, and `--python`, still win over it. `version_source` reports
`env`. To keep endoflife.date's data but fetch it from an internal mirror of
its API v1 response, set `TESTUDOS_ENDOFLIFE_URL`.

#### Refusing a guessed matrix

When neither endoflife.date nor the cache answers, testudos warns and tests its
bundled list. On ephemeral CI runners there is never a cache, so a blocked or
failed request takes that path. `--strict-versions` (config
`strict-versions = true`, environment `TESTUDOS_STRICT_VERSIONS=1`) makes that
an error instead.

#### Which interpreter

`uv run --python=3.12` uses a 3.12 already installed on the machine if there is
one, so the patch release tested depends on the runner image.
`--python-preference only-managed` (config `python-preference`) makes uv use
its own managed builds only, so every machine tests the same release.

Offline mode is also enabled by setting `TESTUDOS_OFFLINE` to anything except
`0`, `false`, `no`, `off`, `n`, `f` or the empty string (case-insensitive, and
surrounding whitespace is ignored). Those values turn it *off*, overriding a
configured `offline = true`; leaving the variable unset says nothing, so the
configuration decides. `--offline` / `--no-offline` win over both.

### Execution

```bash
testudos run --parallel                     # concurrent across versions (-P)
testudos run --parallel --jobs 4            # cap concurrency (-j; needs --parallel)
testudos run --no-parallel                  # override a configured parallel
testudos run --no-fail-fast                 # keep going after a failure (-F)
testudos run --timeout 300                  # seconds per version (-t; default 1800)
testudos run --no-timeout                   # no timeout at all
testudos run --no-setup                     # skip the setup phase (below)
testudos run --setup-timeout 1200           # seconds per version's setup (default 900)
testudos run --verbose                      # more detail (-v)
testudos run --quiet                        # suppress failed-test output (-q)
testudos run --dry-run                      # print commands, run nothing (-n)
```

Ctrl-C, `SIGTERM` and `SIGHUP` all stop a run the same way: every running
test process is interrupted, given five seconds to finish its teardown, and
then killed. testudos exits with code 2. `SIGTERM` is what a cancelled CI
job, `docker stop` or `timeout(1)` sends, so cancelling a pipeline does not
leave suites running behind it.

Progress is a live table when stdout is a terminal. Anywhere else -- a CI
log, a pipe, a file -- testudos prints one line as each version starts and
finishes, and a "still running" line every minute while nothing finishes, so a
long run never looks hung. `FORCE_COLOR` and `--color` add colour to those
lines; they do not turn the live table on.

Every version runs under a timeout, 30 minutes unless you say otherwise
(`timeout = 0` in config, or `--no-timeout`, turns it off), so a hung suite
stops well inside a CI job's limit rather than at it.

Before its tests, each version has a **setup** phase: the same `uv run`, with a
one-line probe in place of the suite. It downloads the interpreter if needed
and installs the environment into uv's cache, under its own timeout, so
`--timeout` measures the tests and a slow first install never eats into it.
The probe also reports the exact interpreter, shown in the results as
`3.12 (3.12.11)` and in JSON as `"python"` -- which patch release a cycle
resolves to depends on the machine. When uv cannot provide the interpreter or
install the environment, the version is reported as `SETUP FAILED`, with what
uv said, and its tests are not run. `--no-setup` (config `setup = false`)
skips the phase.

Fail-fast is on by default in sequential mode. Versions it skips are reported
as `SKIPPED` rather than dropped, so the summary always accounts for every
version you asked for. It does not apply in parallel mode, where every version
is already running by the time the first failure is known.

### Multiple packages

```bash
testudos run --package ./pkg1 --package ./pkg2   # explicit packages (-k)
testudos run ./monorepo --discover               # every dir with a pyproject.toml (-d)
testudos run ./pkg1 --package ./pkg2 --package-jobs 2   # cap package concurrency (-J)
testudos run --package ./pkg1 --package ./pkg2 --no-parallel-packages   # one at a time
```

`--package` names the packages to test. The current directory is included
only when you name it (`testudos run . --package ./other`), so the first
example above tests exactly the two packages it lists.

Package-level and version-level concurrency are independent: `--parallel` /
`--jobs` control versions within a package, `--parallel-packages` /
`--package-jobs` control packages. Each package keeps its own configuration,
results and coverage data.

Those two multiply, so a run is bounded by a third number that does not:

```bash
testudos run ./monorepo --discover --parallel --max-concurrency 4   # (-C)
```

**At most `--max-concurrency` `uv` processes run at any moment, however many
packages and versions there are.** It defaults to one per CPU. `--jobs` and
`--package-jobs` remain per-level caps within it: they can lower a level's
share, never raise the total. Without it, twenty packages across four Python
versions would ask for eighty simultaneous environment builds, which on a
laptop or a 2-core runner shows up as timeouts and OOM kills that look like
flaky tests.

A package's own `max-concurrency` setting applies when it is tested on its
own. In a workspace the packages share one machine, so only `--max-concurrency`
moves the limit.

### Coverage

```bash
testudos run --coverage                                  # collect (-c)
testudos run --coverage --coverage-report html
testudos run --coverage --coverage-report html --coverage-report xml
testudos run --coverage --coverage-fail-under 80
testudos run --coverage --coverage-source mypackage      # default: detected
testudos run --no-coverage                               # override a configured coverage
```

Formats: `term`, `term-missing`, `html`, `xml`, `json`, `lcov`. Data lands in
`.testudos/coverage/`. Without `--coverage-source` (or `coverage-source` in
config), what to measure is read from the project: the packages your build
backend declares, else `src/`, else the importable packages at the project
root. A source that does not exist stops the run before any test starts, since
coverage would measure nothing. The `--coverage-fail-under` threshold is checked
against exact totals, so it gives the same verdict whichever report formats
you asked for.

Tests that run in subprocesses -- pytest-xdist workers, `multiprocessing` --
are measured too, when testudos is installed with the `coverage` extra: the
run is given your project's coverage configuration plus coverage's
`patch = ["subprocess"]`, written to `.testudos/coverage/coveragerc` (edit your
project's own config, not that file). Without the extra only the top process is
measured. Either way, a run that measures 0% of its source says so on stderr.

Coverage is reported whenever data was collected, including when a version
failed its tests. The two verdicts are independent and both feed the exit
code, so a coverage regression cannot hide behind an unrelated test failure.
A failing version still measured the code it ran; one that fail-fast never
reached measured nothing, and the report names the versions it actually
covers.

Coverage data can also be managed directly:

```bash
testudos coverage combine     # merge per-version data
testudos coverage report      # generate a report (--format, --fail-under)
testudos coverage clean       # remove .testudos/coverage
```

Combining keeps the per-version files, so it can be repeated and
`.coverage.3.11` is still there afterwards to answer "which lines does 3.11
miss that 3.13 covers?". `testudos coverage clean` is the only command that
removes coverage data.

### Machine-readable output

```bash
testudos run --format json
testudos run --format json --output results.json    # (-o)
testudos run --format json --dry-run
```

JSON output reports per-version results plus a summary of passed, failed and
skipped counts, and `version_source` says whether the version matrix came from
endoflife.date, the cache or the bundled fallback. With `--coverage` it carries
a `coverage` block (totals, threshold, report paths); in multi-package mode it
has one entry per package under `packages`, each shaped like the
single-package payload. Errors and warnings go to stderr, so stdout is always
one JSON document -- including on failure, when it carries `"success": false`
and an `error`.

With `--output`, the document goes to the file and the terminal gets the same
readable run as without `--format json` -- progress, the results table and the
output of failed versions -- so a CI job can keep a JSON artifact and a log a
person can read.

### Private indexes

```bash
testudos run --default-index https://pypi.internal.example.com/simple
testudos run --index https://extra.example.com/simple
testudos run --find-links /path/to/wheels
testudos run --no-index --find-links /path/to/wheels
testudos run --index-strategy first-index
```

### Colour

```bash
testudos --color run      # force colour
testudos --no-color run   # disable it
```

These are options on `testudos` itself, so they go **before** the subcommand.
`NO_COLOR` and `FORCE_COLOR` are respected too.

### Running in CI

On GitHub Actions (`GITHUB_ACTIONS=true`, which the runner sets) testudos also
writes what GitHub reads from a job:

- each version's output inside a `::group::`, so the log folds to one line
  per version;
- an `::error` annotation for each failed version, setup failure, timeout or
  missed coverage threshold, and a `::warning` for each warning (a fallback
  matrix, an unknown config key) -- these show on the run's summary page and
  the pull request's checks;
- a results table appended to `$GITHUB_STEP_SUMMARY`.

`--ci-format github` turns this on anywhere, `--ci-format none` turns it off.

```bash
testudos run --junit-xml .testudos/junit     # a JUnit XML report per version
```

For pytest, each version writes `.testudos/junit/<version>.xml` itself, with a
suite named `python-<version>`, which test-report actions and GitLab or Jenkins
read. A version pytest did not write one for -- its setup failed, it timed out,
fail-fast skipped it, or the suite is not pytest -- gets a one-test-case report
from testudos, so no version is missing. A relative directory is inside each
package; an absolute one gets a subdirectory per package. The paths are in the
JSON as `"junit_xml"`.

In a monorepo, test only what a change reaches:

```bash
testudos run --discover --changed-since origin/main
```

A package is tested when a file inside it changed since the branch left
`origin/main` (committed, uncommitted or untracked), or when it depends on one
that did -- through `[project].dependencies`, optional dependencies, dependency
groups or `[tool.uv.sources]` -- directly or through other packages. A change
outside every package (a root lockfile, CI configuration) tests all of them.
The rest are listed as not affected, and as `"not_affected"` in JSON; with
nothing affected the run exits 0. It needs the ref's history:
`actions/checkout` with `fetch-depth: 0`.

### Exit codes

| Code | Meaning |
|---|---|
| 0 | Every version passed, and coverage met its threshold |
| 1 | Tests failed, or coverage missed `--coverage-fail-under` |
| 2 | Interrupted: Ctrl-C, `SIGTERM` or `SIGHUP` |
| 3 | testudos could not run: uv or coverage missing, no version data under `--strict-versions`, an environment uv could not install, an unwritable `--output` |
| 4 | The command line or a configuration is wrong: an unknown option or value, an invalid `[tool.testudos]`, a `requires-python` nothing satisfies, a missing coverage source |

A test failure wins over a setup failure in the same run. CI can retry on 3,
send 4 to whoever owns the configuration, and treat 1 as the result.

### Environment variables

Most `run` options can be set from the environment, which is how shared CI
templates are usually configured; a flag on the command line still wins.
`testudos run --help` lists them: `TESTUDOS_TIMEOUT`, `TESTUDOS_PARALLEL`,
`TESTUDOS_JOBS`, `TESTUDOS_MAX_CONCURRENCY`, `TESTUDOS_FAIL_FAST`,
`TESTUDOS_COVERAGE`, `TESTUDOS_COVERAGE_REPORT`, `TESTUDOS_COVERAGE_FAIL_UNDER`,
`TESTUDOS_WITH`, `TESTUDOS_GROUP`, `TESTUDOS_EXTRA`, `TESTUDOS_LOCKED`,
`TESTUDOS_FROZEN`, `TESTUDOS_EXCLUDE_NEWER`, `TESTUDOS_PYTHON_PREFERENCE`,
`TESTUDOS_SETUP`, `TESTUDOS_SETUP_TIMEOUT`, `TESTUDOS_DEFAULT_INDEX`,
`TESTUDOS_INDEX`, `TESTUDOS_FORMAT`, `TESTUDOS_OUTPUT`, `TESTUDOS_PACKAGE_JOBS`.
Repeatable options take a space-separated list (`TESTUDOS_GROUP="test lint"`).
`TESTUDOS_OFFLINE`, `TESTUDOS_STRICT_VERSIONS`, `TESTUDOS_PYTHON_VERSIONS`,
`TESTUDOS_ENDOFLIFE_URL` and `TESTUDOS_TRACEBACK` are described with their
features above.

### Proxies and certificates

testudos makes one HTTP request, to endoflife.date, with Python's standard
library: it honours `HTTPS_PROXY` and `NO_PROXY` and verifies TLS against the
operating system's trust store, so a TLS-inspecting corporate proxy whose
certificate the machine trusts works without extra configuration. uv has its
own settings for the package index (`UV_NATIVE_TLS` / `UV_SYSTEM_CERTS`).

## Configuration

Everything above can be set in `[tool.testudos]`:

```toml
[project]
requires-python = ">=3.11"

[tool.testudos]
# Version selection
python-versions = ["3.11", "3.12", "3.13"]   # override auto-detection
include-prerelease = false                    # include unreleased cycles
offline = false                               # never call endoflife.date
strict-versions = false                       # fail rather than test the bundled list
python-preference = "only-managed"            # uv-managed interpreters only

# What to run
test-command = "pytest"
test-args = ["-v", "--tb=short"]
test-groups = ["test"]                        # [dependency-groups] to install
test-extras = ["test"]                        # extras to install
test-dependencies = ["pytest-asyncio"]        # extra packages per isolated env
timeout = 300                                 # seconds per version (0: none)
setup = true                                  # install each environment first
setup-timeout = 900                           # seconds per version's setup

# The lockfile
locked = true                                 # fail when uv.lock is stale
frozen = false                                # use uv.lock without checking it
exclude-newer = "2026-10-01"                  # ignore releases after this date

# How to run it
junit-xml = ".testudos/junit"                 # a JUnit XML report per version
parallel = false
max-jobs = 4
max-concurrency = 8                           # uv processes at once (default: CPUs)

# Coverage
coverage = false
coverage-combine = true
coverage-report = ["term"]
coverage-fail-under = 80
coverage-source = ["src"]

# Package indexes
default-index = "https://pypi.internal.example.com/simple"
index = ["https://extra.example.com/simple"]
find-links = ["/path/to/wheels"]
no-index = false
index-strategy = "first-index"
```

CLI options win over configuration. Values are validated on load: a wrong
type or an out-of-range value is an error, and an unrecognised key is a
warning.

`test-command` is a command line, not just a program name, so a runner that
needs arguments to name itself fits in one setting:

```toml
test-command = "pytest"
test-command = "python -m unittest discover"
test-command = "./run-tests"
```

It is split the way a shell would split it (quoted words stay together), but
no shell runs it: the pieces become an argv list. Only the first word is a
package name, so it is what gets installed into each isolated environment —
a suite run as `python -m pytest` should name `pytest` in `test-dependencies`,
since the interpreter is what testudos sees. A first word containing a `/`,
like `./run-tests`, is a file in your project rather than a package, so
nothing is installed for it.

`test-args` are passed to that command exactly as written, with no filtering,
so metacharacters need no escaping and no workaround:

```toml
test-args = ["--ignore-glob=*$py.class", "-W", "error:.*deprecated$:UserWarning"]
```

By the same token nothing in them is expanded either — `--rootdir=$PWD`
arrives as the literal string `$PWD`, not as your working directory.

## Documentation

- [Architecture](docs/ARCHITECTURE.md) — how the pieces fit together, and the decisions behind them

## Development

```bash
git clone https://github.com/martinristovski/testudos.git
cd testudos
uv sync --extra dev

just ci            # lint, type-check, test, network tests, dogfood, version check
just test          # pytest with coverage
just fix           # auto-fix lint and formatting
just dogfood       # run testudos on itself
```

The default test suite is hermetic: the endoflife.date API is stubbed and the
version cache is redirected per test, so no test touches the network or your
real `~/.cache/testudos`.

Seven tests do reach the network: six exercise the real `uv` boundary, and so
resolve packages from PyPI, and one checks the canned endoflife.date response
against the live API. They are marked `network` and deselected by default; run
them with `pytest -m network`. CI runs them as their own step, so an index or
API outage fails a step that names the cause.

## License

MIT
