Metadata-Version: 2.4
Name: pytest-respect
Version: 1.0.0
Summary: Pytest plugin to load resource files relative to test code and to expect values to match them.
Project-URL: homepage, https://github.com/Ankeri/pytest-respect
Project-URL: issues, https://github.com/Ankeri/pytest-respect/issues
Author-email: Logi Ragnarsson <logi.ragnarsson@ankeri.is>
License-Expression: MIT
Classifier: Development Status :: 5 - Production/Stable
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: CPython
Requires-Python: >=3.12
Requires-Dist: pytest>=8.0.0
Provides-Extra: jsonyx
Requires-Dist: jsonyx>=2.0.0; extra == 'jsonyx'
Provides-Extra: numpy
Requires-Dist: numpy>=2.0.0; extra == 'numpy'
Provides-Extra: pydantic
Requires-Dist: pydantic>=2.0.0; extra == 'pydantic'
Description-Content-Type: text/markdown

# pytest-respect

Pytest plugin to load resource files relative to test code and to expect values to match such files. The name is a contraction of `resources.expect`, which is frequently typed when using this plugin.

## Motivation

The primary use-case is running tests over moderately large datasets where adding them as constants in the test code would be cumbersome. This happens frequently with integration tests or when retrofitting tests onto an existing code-base. If you find your test _code_ being obscured by the test _data_, filling with complex data generation code, or ad-hoc reading of input data or expected results, then pytest-respect is for you.

What differentiates pytest-respect from similar libraries is that comparisons are made on text dumps of your data, rather than constructing complex comparators. It has native integration for dumping pydantic and numpy data to JSON and an easy way to customize dumping of other data types.

## Installation

Install with your favourite package manager such as:

- `pip install pydantic-respect`
- `poetry add --dev pydantic-respect`
- `uv add --dev pydantic-respect`

See your package management tool for details, especially on how to install optional extra dependencies.

### Extras

Add the following extra dependencies for additional functionality:

- `poetry` - Load, save, and expect pydantic models or arbitrary data through type adapters.
- `numpy` - Convert numpy arrays and scalars to python equivalents when generating JSON, both in save and expect.
- `jsonyx` - Alternative JSON encoder for semi-compact files, numeric keys, trailing commas, etc.

## Usage

### Text Data

The simplest use-case is loading textual input data and comparing textual output to an expectation file:

```python
def test_translate(resources: TestResources):
    input = resources.load_text("input")
    output = translate(input)
    resources.expect_text(output, "output")
```

If the test is found in a file called `foo/test_stuff.py`, then it will load the content of `foo/test_stuff/test_translate__input.txt`, run the `translate` function on it, and assert that the output exactly matches the content of the file `foo/test_stuff/test_translate__output.json`.

The expectation must also match on trailing spaces and trailing empty lines for the test to pass.

### Json Data

A much more interesting example is doing the same with JSON data:

```python
def test_compute(resources: TestResources):
    input = resources.load_json("input")
    output = compute(input)
    resources.expect_json(output, "output")
```

This will load the content of `foo/test_stuff/test_compute__input.json`, run the `compute` function on it, and assert that the output exactly matches the content of the file `foo/test_stuff/test_compute__output.json`.

The expectation matching is done on a text representation of the JSON data. This avoids having to parse the expectation files, and allows us to use text-based diff tools, but instead we must avoid other tools reformating the expectations. By default the JSON formatting is by `json.dumps(obj, sort_keys=True, indent=2)` but see the section on [JSON Formatting and Parsing](#json-formatting-and-parsing).

### Pydantic Models and Type Adapters

With the optional
`pydantic` extra, the same can be done with pydantic data if you have models for your input and output data:

```python
def test_compute(resources: TestResources):
    input: InputModel = resources.load_pydantic(InputModel, "input")
    output: OutputModel = compute(input)
    resources.expect_pydantic(output, "output")
```

The input and output paths will be identical to the JSON test, since we re-used the name of the test function.

There are also `load_pydantic_adatper` and `expect_pydantic_adapter` variants which take a pydantic `TypeAdapter` instead of a model class, or they can take an arbitrary type to wrap in a `TypeAdapter` instance. Please refer to the pydantic documentation for more information on type adapters.

### Failing Tests

#### Actual Files

If an expectation fails, then a new file is created containing the actual value passed to the expect function. Its path is constructed in the same way as that of the expectation file, but with an `actual` part appended. In the JSON and Pydantic examples above, it would create the file `foo/test_stuff/test_compute__output__actual.json`. In addition to this, the normal pytest assert re-writing is done to show the difference between the expected value and the actual value.

When the values being compared are large or complex, the difference shown on the console may be overwhelming. Then you can instead use your existing diff tools to compare the expected and actual files and perhaps pick individual changes from the actual file before fixing the code to deal with any remaining differences.

Once the test passes, the `__actual` file will be removed. Note that if you change the name of a test after an actual file has been created, then it will have to be deleted manually.

#### Accepting Changes

Alternatively, if you know that all the actual files from a test run are correct, you can run the test with the `--respect-accept` flag to update all the expectations. You can also use the `--respect-accept-one` and `--respect-accept-max=n` flags to update only a single expectation or the first `n` expectations for each test, before failing on any remaining differences.

### Resource Path Construction

#### Multiple name parts

In all of the above examples, we passed a single string `"input"` or `"output"` to the load or expect methods. We can pass as many such name parts as we like, which affects the name of the resource file.

Using the JSON and Pydantic examples above, these paths would be constructed:

- `resources.load_json()` → `foo/test_stuff/test_compute.json`
- `resources.load_json("data")` → `foo/test_stuff/test_compute__data.json`
- `resources.load_json("scenario", "funky")` → `foo/test_stuff/test_compute__scenario__funky.json`

#### Path Makers

So far all our resource paths have been fairly rigidly constructed from the path to the test file and the test function within it. The way this is done is in fact fully configurable by passing a custom `PathMaker` to any method which accesses resource files, or by assigning a different one to `resources.default.path_maker`. A path maker is any function which implements the `PathMaker` protocol and a few standard ones are already present on the `resources` fixture.

If we revisit the JSON example from above, but using a different path maker, it will function in exactly the same way except that the resource files will be at `foo/test_stuff/input.json` and `foo/test_stuff/output.json` instead, ignoring the test function name.

```python
def test_compute(resources: TestResources):
    input = resources.load_json("input", path_maker=resources.pm_only_file)
    output = compute(input)
    resources.expect_json(output, "output", path_maker=resources.pm_only_file)
```

The same test can instead be written by setting the default path_maker with:

```python
def test_compute(resources: TestResources):
    resources.default.path_maker = resources.pm_only_file
    input = resources.load_json("input")
    output = compute(input)
    resources.expect_json(output, "output")
```

The table below shows the paths made by the different path makers when calling `resource.path("data")` in a `test_function` in a `test_file.py` in `<dir>`, or if `test_function` is a member of a `TestClass`.

| Path Maker                      | **test_function**<br/> **TestClass.test_method**
|---------------------------------|--------------------------------------------------------------------------------------------------------------------------
| `pm_function`                   | `<dir>/test_file__test_function/data.ext`            <br/>`<dir>/test_file__TestClass__test_method/data.ext`
| `pm_class`                      | `<dir>/test_file/test_function__data.ext`            <br/> `<dir>/test_file__TestClass/test_method__data.ext`
| `pm_only_class`                 | `<dir>/test_file/data.ext`                           <br/> `<dir>/test_file__TestClass/data.ext`
| `pm_file`                       | `<dir>/test_file/test_function__data.ext`            <br/> `<dir>/test_file/TestClass__test_method__data.ext`
| `pm_only_file`                  | `<dir>/test_file/data.ext`                           <br/> `<dir>/test_file/data.ext`
| `pm_dir`                        | `<dir>/resources/test_file__test_function__data.ext` <br/> `<dir>/resources/test_file__TestClass__test_method__data.ext`
| `pm_dir_named("dir_name")`      | `<dir>/dir_name/test_file__test_function__data.ext`  <br/> `<dir>/dir_name/test_file__TestClass__test_method__data.ext`
| `pm_only_dir`                   | `<dir>/resources/data.ext`                           <br/> `<dir>/resources/data.ext`
| `pm_only_dir_named("dir_name")` | `<dir>/dir_name/data.ext`                            <br/> `<dir>/dir_name/data.ext`

#### Custom Path Makers

If none of these strategies suits your needs, then you can make your own path maker with the same signature as one of the included ones and use that instead.

The following example is similar to the default `pm_file` path maker, but creates a sub-directory for each date inside the resource directory:

```python
def pm_file_dated(test_dir: Path, test_file_name: str, test_class_name: str | None, test_name: str) -> PathParts:
    file = f"{test_class_name}__{test_name}" if test_class_name else test_name
    sub = date.today().isoformat()
    return test_dir / sub / test_file_name, file
```

### Other I/O on Resource Files

Each of the `load` and `expect` methods above also has a corresponding `save` method which simply writes the data to a file, as well as a `delete` method. The resource path resolution is also exposed as the `resources.path(*parts: str, ext: str | None = None, path_maker: PathMaker | None = None)` method if you need to access the files directly.

Using those, we can test a function which manilpulates external data files:

```python
def test_external_processing(resources: TestResources):
    resources.save_json(make_data(), "data")
    path: Path = resources.path("data", ext="json")
    result = external_processing(path)
    assert result == 42
    resources.delete_json("data")
```

As a utility, the `save_foo` and `delete_foo`methods also return the path to the affected file, so the test can be written as:

```python
def test_external_processing(resources: TestResources):
    path: Path = resources.save_json(make_data(), "data")
    result = external_processing(path)
    assert result == 42
    path.unlink()
```

Finally, the `resources.list()` method lists the names of resources within the test's resource folder as constructed by the path maker. It takes one or more include or exclude glob patterns to filter the results and defaults to `inclue="*"` with no `exclude`.

```python
def test_compute(resources: TestResources):
    widget_names: list[str] = resources.list("widget_*.json", strip_ext=True)
    for widget_name in widget_names:
        widget = resources.load_json(widget_name)
        assert transform(widget) == 42
```

Depending on the rest of your test, you may want to exclude the `*__actual.json` files which might have been created in a previous test run.

This tetst also has the problem that if the assert fails on one file, then the test terminates and we won't know if this is an isolated problem of affects more files. See the section on  [Data-driven Parametric Tests|#data-driven-parametric-tests] for a solution to that.

### Parametric Tests

We have seen how the `load` and `expect` (and other) methods can take multiple strings for the resource file name `parts`. In the earlier examples we only used `"input"` and `"output"` parts and failures implicitly added an `"actual"` part. But using multiple parts is useful when working with parametric tests:

```python
@pytest.mark.paramtrize("case", ["red", "green", "blue"])
def test_compute(resources, case):
    input = resources.load_json(case, "input")
    output = compute(input)
    resources.expect_json(output, case, "output")
```

Omitting the directory name, this test will load each of `test_compute__red__input.json`, `test_compute__green__input.json`, `test_compute__blue__input.json` and compare the results to `test_compute__red__output.json`, `test_compute__green__output.json`, `test_compute__blue__output.json`

### Data-driven Parametric Tests

We can use the `list_resources` function to generate a list of resource names to run parametric tests over. With the below fixture, the content of the resource directory is listed, and the fixture is run once for each match. We can then add test cases simply by adding new resource files:

```python
@pytest.fixture(params=list_resources("widget_*.json", strip_ext=True))
def each_widget_name(request) -> str:
    """Request this fixture to run for each widget file in the resource directory."""
    return request.param
```

The `list_resources` function is run in a static context and so doesn't have a test function or class to build paths from. Instead, it constructs a path to the file that it is called from and uses the `pm_only_file` path maker by default. However, it takes an optional `path_maker` argument to override this.

Tests can then request `each_widget_name` to run on each of the resources but will have to use a suitable path-maker to find the resource files:

```python
def test_load_json_resource(resources, each_widget_name):
    widget = resources.load_json(each_widget_name, path_maker=resources.pm_only_file)
    assert transform(widget) == 42
```

### JSON Files

#### Deterministic Output

Since `expect_json`and `expect_pydantic` both ultimately create a JSON text representation of the data, which then gets compared with an expectation file on disk, the generated files need to be predictable. Much of that is up to the developer, to make sure that their code always generates the same output for the same input. In certain cases that's not possible or feasible, and then these external libraries may be of help:

- [pytest-freezer](https://pypi.org/project/pytest-freezer/) to freeze time and return the same timestamps in each test run.
- [pytest-randomly](https://pypi.org/project/pytest-randomly/) to freeze random number generators and return the same random numbers in each test run.
- [pytest-mock](https://pypi.org/project/pytest-mock/) to mock out any non-deterministic functions called by the code under test.

#### Formatting

JSON formatting is done with a JSON-encoder, which is a simple function converting data to a string. By default this is `python_json_encoder` which just wraps a call to `json.dumps(obj, sort_keys=True, indent=2)` for a standard verbose JSON dump.

You can change the encoding by passing a `json_encoder` argument to any function which writes JSON files or compares data to such files or by assigning a different one to `resources.default.json_encoder`.

The following encoders are included with the library, but the ones with `jsonyx` in the name require the `jsonyx` extra dependency to be installed:

| JSON Encoder                  | Properties
|-------------------------------|--------------------------------------------------------------------
| `python_json_encoder`         | Standard JSON encoder in very verbose mode
| `python_compact_json_encoder` | Standard JSON encoder in very compact mode
| `jsonyx_encoder`              | JSONYX encoder which allows non-string dict keys
| `jsonyx_compactish_encoder`   | Like `jsonyx_encoder` but with deepest nested structure unindented
| `jsonyx_compact_encoder`      | Like `jsonyx_encoder` but in very compact mode

#### Data Prepping

In the Formatting section there is no talk of dealing with data which is not normally JSON encodeable. This functionality is separated out into "JSON Preppers" which know how to convert certain types of data into JSON-encodeable forms. At launch time, an attempt is made to install preppers for Pydantic models and numpy arrays if the corresponding libraries are found.

You can register your own global preppers by calling `pytest_respect.utils.add_json_prepper(type, prepper)` or a short-lived one by calling `resources.add_json_prepper(type, prepper)`.

When preparing a particular value for JSON encoding, a search is made for a prepper for that type (or a super-type thereof), starting with the ones added on the `resources` fixture and then continuing on to the global ones. When a prepper is found, it is called with the value and the result is used in its place, unless the prepper raises the `AbortJsonPrep` exception, in which case that prepper is ignored and the search continues.

If a value is converted into a dict, list or tuple, then the preparation continues recursively inside the converted value so that you can prepare fields nested deep within pydantic models or other mutually reursive types.

#### Loading

JSON loading is done with a JSON-loader, which is a simple function converting a string to data. By default this is `python_json_loader` which just wraps a call to `json.loads(text)` for standard JSON parsing.

You can change the loading by passing a `json_loader` argument to any function which reads JSON files or by assigning a different one to `resources.default.json_loader`.

The following loaders are included with the library, but the one prefixed with `jsonyx` requires the `jsonyx` extra dependency to be installed:

| JSON Loader                | Properties
|----------------------------|--------------------------------------------------------------------
| `python_json_loader`       | Standard JSON loader
| `jsonyx_permissive_loader` | JSONYX loader which allows non-string dict keys and other extensions

### Configuration

You can configure default behavior of the `resources` fixture in your project by overriding in any particular scope. For global configuration, do this in your project's root `conftest.py` file. This allows you to set defaults for path makers, JSON encoders and loaders, and float rounding without having to specify them in every test.

```python
@pytest.fixture
def resources(resources: TestResources) -> TestResources:
    """Configure resources for the scope of this fixture."""

    resources.default.ndigits = 4
    resources.default.json_encoder = python_json_encoder
    resources.default.json_loader = python_json_loader
    resources.default.path_maker = resources.pm_file

    return resources
```

The available configuration options are:

- `resources.default.ndigits` - How many digits to round floats to when comparing JSON data. Defaults to `None` to disable rounding.
- `resources.default.json_encoder` - Function used to convert data to JSON encoded text (default: `python_json_encoder`).
- `resources.default.json_loader` - Function used to convert JSON encoded text to python data (default: `python_json_loader`).
- `resources.default.path_maker` - Function used to construct paths to resource files (default: `pm_file`).

## Development

### Installation

- [Install uv](https://docs.astral.sh/uv/getting-started/installation/)
- Run `uv sync --all-extras`
- Run `pre-commit install` to enable pre-commit linting.
- Run `pytest` to verify installation.

### Testing

This is a pytest plugin so you're expected to know how to run pytest when hacking on it. Additionally, `scripts/pytest-extras` runs the test suite with different sets of optional extras. The CI Pipelines will go through an equivalent process for each Pull Request.
