# Auto Test

PlestyLib includes lightweight auto-test helpers to quickly validate a device API implementation.

These helpers are intended for development-time smoke testing, not full regression testing.

## Available Auto Tests

There are two utilities:

1. Config auto test: validates registered config parameters (read-only, write-only, read-write).
2. Function auto test: validates registered operations in the function system.

## 1) Config Auto Test (Synchronous Device API)

Utility location:

`plestylib.test.auto_test_sync_device.auto_test`

What it does:

1. Opens the device with context manager.
2. Prints identity.
3. Queries all read-only configs.
4. Writes all write-only configs using defaults or dtype-based fallback values.
5. Writes and re-queries read-write configs.

Example:

```python
from plestylib.test.auto_test_sync_device import auto_test
from powermeter_device import PowermeterDevice


auto_test(
	 PowermeterDevice,
	 "USB0::0x1313::0x8078::P0000001::INSTR",
	 sensor_type="S155C",
)
```

Notes:

1. For read-write parameters without default value, current queried value is written back.
2. This avoids guessing unsupported values when device constraints are incomplete.

## 2) Function Auto Test (Operation/Func System)

Utility location:

`plestylib.test.auto_test_func_system.auto_test`

What it does:

1. Opens the device and optionally queries identity.
2. Enumerates registered operations from `device._functions`.
3. Generates valid payloads from `FuncParam` metadata.
4. Calls each operation and checks response shape.
5. Prints pass/fail summary.

By default it can use a mock operation solver so operation validation can run without real backend behavior.

Example (mock solver mode):

```python
from plestylib.test.auto_test_func_system import auto_test
from my_device import MyDevice


auto_test(
	 MyDevice,
	 "device-id-or-address",
	 use_mock_solver=True,
	 seed=123,
)
```

Example (real solver mode):

```python
from plestylib.test.auto_test_func_system import auto_test
from my_device import MyDevice


auto_test(
	 MyDevice,
	 "device-id-or-address",
	 use_mock_solver=False,
	 ignore_ops=["dangerous_operation"],
)
```

## Important Parameters (Function Auto Test)

1. `use_mock_solver`:
	When true, binds an internal mock solver that generates output-shaped responses from metadata.
2. `ignore_ops`:
	List of operation names to skip.
3. `seed`:
	Seed for deterministic test payload generation.
4. `sleep_time`:
	Delay between operation calls.

## Output Interpretation

Function auto test prints one line per operation:

1. `[PASS] op_name payload=... response=...`
2. `[FAIL] op_name payload=... error=...`

Then a summary:

1. Passed count
2. Failed count
3. Passed operation list

## Recommended Workflow

1. Start with config auto test after registering parameters.
2. Add function auto test after registering operations.
3. Run function auto test first with `use_mock_solver=True` to validate schemas.
4. Then switch to `use_mock_solver=False` for end-to-end backend checks.
5. Add operation names to `ignore_ops` for stateful or destructive actions.

## Limitations

1. These helpers are print-based and not pytest assertions by default.
2. They do not replace integration tests for timing, hardware state transitions, or long-running sequences.
3. Mock mode validates interface contracts, not real protocol/device behavior.

## Using with pytest

You can leverage auto-test utilities inside `pytest` suites in two ways.

### 1. Smoke-test style (no exception means pass)

```python
from plestylib.test.auto_test_sync_device import auto_test as auto_test_sync
from plestylib.test.auto_test_func_system import auto_test as auto_test_func
from my_device import MyDevice


def test_config_auto():
    auto_test_sync(MyDevice, "device-id-or-address")


def test_function_auto_mock():
    auto_test_func(MyDevice, "device-id-or-address", use_mock_solver=True, seed=123)
```

### 2. Assertion style with pytest output capture

Because these helpers currently print summary lines, use pytest `capsys` to capture stdout and assert summary text.

```python
from plestylib.test.auto_test_func_system import auto_test
from my_device import MyDevice


def test_no_failed_operations(capsys):
    auto_test(MyDevice, "device-id-or-address", use_mock_solver=True, seed=123)
    captured = capsys.readouterr()
    assert "Failed: 0" in captured.out, captured.out
```

### Recommendation

For stronger tests, consider extending auto-test helpers to return a summary dictionary (for example passed/failed counts) in addition to printing. Then `pytest` assertions can be done directly without parsing text output.
