Auto Test

PlestyLib includes lightweight auto-test helpers to quickly validate a device API implementation.

These helpers are intended for development-time smoke testing, not full regression testing.

Available Auto Tests

There are two utilities:

1. Config auto test: validates registered config parameters (read-only, write-only, read-write).

2. Function auto test: validates registered operations in the function system.

1) Config Auto Test (Synchronous Device API)

Utility location:

plestylib.test.auto_test_sync_device.auto_test

What it does:

1. Opens the device with context manager.

2. Prints identity.

3. Queries all read-only configs.

4. Writes all write-only configs using defaults or dtype-based fallback values.

5. Writes and re-queries read-write configs.

Example:

from plestylib.test.auto_test_sync_device import auto_test
from powermeter_device import PowermeterDevice


auto_test(
	 PowermeterDevice,
	 "USB0::0x1313::0x8078::P0000001::INSTR",
	 sensor_type="S155C",
)

Notes:

1. For read-write parameters without default value, current queried value is written back.

2. This avoids guessing unsupported values when device constraints are incomplete.

2) Function Auto Test (Operation/Func System)

Utility location:

plestylib.test.auto_test_func_system.auto_test

What it does:

1. Opens the device and optionally queries identity.

2. Enumerates registered operations from device._functions.

3. Generates valid payloads from FuncParam metadata.

4. Calls each operation and checks response shape.

5. Prints pass/fail summary.

By default it can use a mock operation solver so operation validation can run without real backend behavior.

Example (mock solver mode):

from plestylib.test.auto_test_func_system import auto_test
from my_device import MyDevice


auto_test(
	 MyDevice,
	 "device-id-or-address",
	 use_mock_solver=True,
	 seed=123,
)

Example (real solver mode):

from plestylib.test.auto_test_func_system import auto_test
from my_device import MyDevice


auto_test(
	 MyDevice,
	 "device-id-or-address",
	 use_mock_solver=False,
	 ignore_ops=["dangerous_operation"],
)

Important Parameters (Function Auto Test)

1. use_mock_solver: When true, binds an internal mock solver that generates output-shaped responses from metadata.

2. ignore_ops: List of operation names to skip.

3. seed: Seed for deterministic test payload generation.

4. sleep_time: Delay between operation calls.

Output Interpretation

Function auto test prints one line per operation:

1. [PASS] op_name payload=... response=...

2. [FAIL] op_name payload=... error=...

Then a summary:

1. Passed count

2. Failed count

3. Passed operation list

Recommended Workflow

1. Start with config auto test after registering parameters.

2. Add function auto test after registering operations.

3. Run function auto test first with use_mock_solver=True to validate schemas.

4. Then switch to use_mock_solver=False for end-to-end backend checks.

5. Add operation names to ignore_ops for stateful or destructive actions.

Limitations

1. These helpers are print-based and not pytest assertions by default.

2. They do not replace integration tests for timing, hardware state transitions, or long-running sequences.

3. Mock mode validates interface contracts, not real protocol/device behavior.

Using with pytest

You can leverage auto-test utilities inside pytest suites in two ways.

1. Smoke-test style (no exception means pass)

from plestylib.test.auto_test_sync_device import auto_test as auto_test_sync
from plestylib.test.auto_test_func_system import auto_test as auto_test_func
from my_device import MyDevice


def test_config_auto():
    auto_test_sync(MyDevice, "device-id-or-address")


def test_function_auto_mock():
    auto_test_func(MyDevice, "device-id-or-address", use_mock_solver=True, seed=123)

2. Assertion style with pytest output capture

Because these helpers currently print summary lines, use pytest capsys to capture stdout and assert summary text.

from plestylib.test.auto_test_func_system import auto_test
from my_device import MyDevice


def test_no_failed_operations(capsys):
    auto_test(MyDevice, "device-id-or-address", use_mock_solver=True, seed=123)
    captured = capsys.readouterr()
    assert "Failed: 0" in captured.out, captured.out

Recommendation

For stronger tests, consider extending auto-test helpers to return a summary dictionary (for example passed/failed counts) in addition to printing. Then pytest assertions can be done directly without parsing text output.