Metadata-Version: 2.4
Name: pytest_harness
Version: 0.2.0
Summary: Easy-to-use pytest workflow orchestrator with isolated test-file execution, combined coverage, dashboard summaries, and optional per-test-file logs.
Author-email: "R. Wasson" <r.wasson.dev@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/rwasson/pytest_harness
Project-URL: Repository, https://github.com/rwasson/pytest_harness
Project-URL: Issues, https://github.com/rwasson/pytest_harness/issues
Keywords: pytest,testing,coverage,test runner,python,IDE
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Framework :: Pytest
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: logduo>=0.2.0
Requires-Dist: pytest>=9.0.0
Requires-Dist: pytest-cov>=5.0.0
Requires-Dist: pytest-json-report
Requires-Dist: pytest-rerunfailures<17,>=16
Requires-Dist: pytest-timeout
Requires-Dist: pytest-random-order
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: twine>=7.0.0; extra == "dev"
Requires-Dist: ruff>=0.16.2; extra == "dev"
Requires-Dist: mypy<3,>=1.10; extra == "dev"
Requires-Dist: vulture; extra == "dev"
Requires-Dist: pre-commit>=4.6.1; extra == "dev"
Dynamic: license-file

pytest_harness
==============

An easy-to-use Python testing workflow orchestrator built around pytest.

Set the project paths and call `pytest_harness()` in a small runner script.
pytest_harness automatically manages isolated test execution, combined coverage,
dashboard output, summary logging, and optional per-test-file logs.

The runner script can be launched with one click from an IDE or run like any
other Python script from Terminal.


Why use pytest_harness?
-----------------------
- no command-line flags, `pyproject.toml`, coverage settings, or logging setup required
- compact color-coded dashboard and plain-text log that report:
    - test file progress
    - summary of test results
    - list of test files with processing errors and warnings
    - randomization seed, timeout limit, and slow test functions (if any)
    - combined coverage with optional coverage for each source file
    - output directory and names of created log files
- test files run independently; if one test file crashes or fails during import,
  collection, or execution, pytest_harness records the problem and continues to the next test file
- optional per-test-file logs containing pytest results, warnings, and `print()` output
    - support visual inspection of test and function outputs for debugging
    - display missing (uncovered) source lines
    - can serve as lightweight documentation of tested behavior and passing status
- detailed docstring with complete argument list:  

        from pytest_harness import pytest_harness

        help(pytest_harness)

pytest_harness is a workflow orchestrator, not a pytest plugin.

    
Quick Start: Example runner
--------------------------
    """
    pytest_harness_runner.py
    """
    import sys
    from dataclasses import fields
    from pathlib import Path

    from pytest_harness import pytest_harness
    
    # --- Path settings ---
    PROJECT_ROOT = Path(__file__).resolve().parents[1]
    
    # pytest_harness creates the dashboard and logs automatically.
    # The returned results are optional and support additional analysis.
    results = pytest_harness(
        test_file_dir=PROJECT_ROOT / "tests",
        log_dir=PROJECT_ROOT / "tests" / "logs",
        tested_code_dir=PROJECT_ROOT / "src" / "sample_package",
        log_keep=3,
        dashboard_width=80,
        console_theme="light",
        random_order=True,
        test_function_timeout=2,
        show_skipped_and_xfailed=True,
    )

    # --- Inspect returned results --- 
    print("\nFields in results")
    print("-----------------")
    for field in fields(results):
        print(field.name)
    
    # test_function_records is useful for repeated-run and flaky-test analysis
    print("\nFirst 3 results.test_function_records")
    print("-------------------------------------")
    for record in results.test_function_records[:3]:
        print(f"{record}\n")

    # --- Exit with pytest_harness_exit code if any test problems ---
    if results.pytest_harness_exit_code != 0:
        sys.exit(results.pytest_harness_exit_code)


Example console output
----------------------
Console output is color-coded (logs are plain text).

    Running 7 test files: ....... done
    
    Pytest Harness Summary
    ════════════════════════════════════════
    Random-order seed: 1002631522
    Test-function timeout (in seconds): 2.0
    
    
    Test file summary
    ----------------------------------------
    Source files covered:         2
    Test files run:               7
    Test files passed all tests:  4
    
    Test files not processed successfully (1):
        Test function timed out (1):
            test_timeout.py
                    in: test_02_exceeds_test_function_timeout
    
    Test files with no collected tests (1):
        test_empty.py
    
    Test files with warnings (details in individual test-file logs) (1):
        unit_tests/test_intentional_warning.py (1 warning)

    Test function summary
    ----------------------------------------
        Passed:     25
        Failed:      1
        Error:       1
        XPassed:     1
        Skipped:     1
        XFailed:     1
    
    Flagged test functions (in 1 test file):
        unit_tests/test_flagged.py
            Failed (1):
                test_03_intentional_fail
            Error (1):
                test_06_intentional_error_during_setup
            XPassed (1):
                test_05_intentional_unexpected_pass
            Skipped (1):
                test_02_intentional_skip
            XFailed (1):
                test_04_intentional_expected_failure

    Total coverage
    ----------------------------------------
        Statements:   83%
        Branches:     58%
        Total:        76%
    
    WARNING: Total coverage (76%) is below recommended threshold 85%.
    
    Source file   Executed/    Source
    coverage      statements   file
    ------------  -----------  --------------
    55%           6/11         calculator.py
    100%          19/19        helpers/arithmetic_functions.py

    Slowest test functions
    ----------------------------------------
    0.51 s  examples/basic_project/tests/unit_tests/test_flagged.py::test_07_slow_
                test
    
    
    Pytest Harness exit code: 1 (test run errors detected)
    ───────────────────────────────────────────────────────
    Pruned run dirs:  1
    Logging ended:    2026-08-08 10:12:13 (duration 04 sec)
    Script path:
        /Users/<my_name>/basic_project/tests/pytest_harness_runner.py
    Output directory:
        /Users/<my_name>/basic_project/tests/logs/pytest_harness_runner/run_2026_08_08_
            _10_12_09
    Log-generated files in output directory:
        pytest_harness_runner.log
        test_calculator.log
        ...


Arguments
---------
- **test_file_dir**: `pathlib.Path`  
   Existing directory containing pytest test files.

- **log_dir**: `pathlib.Path`  
    Directory where pytest_harness creates run logs. The directory is created
    if it does not already exist.

- **tested_code_dir**: `pathlib.Path`  
    Existing dedicated directory containing the code targeted by the tests and
    measured for coverage. Its parent is used for imports in each pytest
    subprocess.

- **include_list**: `list[str | pathlib.Path] | None`  
    Optional test files or directories to run. Default is None, which discovers
    all matching test files under `test_file_dir`.

- **exclude_list**: `list[str | pathlib.Path] | None`  
    Optional test files or directories to exclude after discovery or inclusion.
    Default is None.

- **random_order**: `bool`  
    Randomize test-function execution order. Default is False.

- **random_order_seed**: `int | None`  
    Seed used when `random_order=True`. If None, pytest_harness generates and
    reports a seed so the test order can be reproduced. Default is None.

- **test_function_timeout**: `int | float | None`  
    Maximum execution time in seconds for an individual test function.
    Use None to disable the timeout. Default is 200.

- **runner_log**: `bool`  
    Write the main pytest_harness summary log. Default is True.

- **test_file_logs**: `bool`  
    Write one detailed log for each selected test file. Default is True.

- **console_dashboard**: `bool`  
    Display the summary dashboard in the console. Default is True.

- **coverage_warning_threshold**: `float | None`  
    Warn when total coverage is below this percentage. This does not affect the
    process exit code. Default is 85.0. Use 0 or None to disable the warning.

- **show_source_file_coverage**: `bool`  
    Display the source-file coverage table. Default is True.

- **show_skipped_and_xfailed**: `bool`  
    Include Skipped and XFailed outcomes in flagged-test section. Default is False.

- **log_keep**: `int | None`  
    Number of recent marked run directories to retain. Default is None, which
    disables pruning.

- **console_theme**: `str`  
    Console color theme: `"dark"` or `"light"`. Default is `"dark"`.

- **dashboard_width**: `int`  
    Width used for the console and log dashboard. Must be at least 80.
    Default is 150.

- **debug_harness**: `bool`  
    Display additional internal diagnostics, including selected test files and
    captured output from flagged test-file subprocesses. Default is False.

Return value
------------
`pytest_harness()` returns structured run, test-file, test-function, and coverage
results for optional programmatic analysis.
The dashboard and logs are generated automatically; using the returned results
is not required.

For complete validation rules and documentation:

    from pytest_harness import pytest_harness
    help(pytest_harness)


Pytest Harness Exit Codes
-------------------------
pytest_harness displays an exit code and includes it in the returned results:

    0    no test run errors detected
    1    test run errors detected

Any of the following produce `pytest_harness_exit_code = 1`:
- one or more test functions Failed
- one or more test functions produced Error
- one or more tests XPassed
- one or more selected test files could not be processed
- one or more selected test files collected no tests

Note:
- Skipped and XFailed outcomes do not by themselves cause exit code 1.
- They also do not trigger tests being listed under `Flagged test functions` 
  unless `show_skipped_and_xfailed=True` (default = False)


Tips
----
- Runner for pytest_harness(): suggested name is `pytest_harness_runner.py`
  - Place pytest_harness_runner.py in your project's test directory.
  - Place test_helper files in a subdirectory inside your project's test directory.
  - pytest_harness uses Logduo internally for its own console output and test-run
    logs. Do not add separate logging setup for the runner or for
    pytest_harness-generated test-file logs. 
  - If desired, set up logging in your runner after calling pytest_harness().
  - Give your runner script a name that does not start with
    `test_` or end with `_test.py`. PyCharm may treat those names as pytest test
    files rather than as executable runner scripts (right-click won't work).

- Test files:
  - Test file names must match `test_*.py` or `*_test.py`.
  - Test function names must begin with `test_`.
  - Suggestion: give test functions names that are descriptive and include a 
    numeric component to make it easier to quickly find and edit flagged tests, 
    (e.g., `test_01_verify_add_calculation_and_output_to_console_and_log`).
  - Keep test functions independent and do not rely on test function execution order.
  - Use pytest's `tmp_path` fixture when tests create temporary files or directories 
    (see example test file below).
  - `print()` statements inside a test file are captured in the test file's 
    individual log and can be used to verify the tested code is executing as intended.
  - Test functions may configure and inspect Logduo logging via Duo() when logging 
    behavior is part of what the test verifies (see example test file below).

    
Example test file
-----------------
    """
    test_calculator.py

    When debugging a failing test,
    place diagnostic print statements before the assertion being investigated.
    Execution of that test function stops when an assertion fails.
    """
    from pathlib import Path

    import pytest
    from logduo import Duo
    from sample_package.calculator import calculation_report
    from sample_package.helpers.arithmetic_functions import add

    def test_01_verify_add_calculation_and_output_to_console_and_log(
        tmp_path: Path,
        capsys: pytest.CaptureFixture[str],
    ) -> None:

        assert add(4, 7) == 11
    
        # Duo() creates an isolated Logduo instance local to this test function.
        # log.close() closes its logging session.
        # pytest provides the tmp_path directory and removes it after the test run.
        log_dir_path = str(tmp_path / "logs")
        log = Duo()
        log.configure(log_dir_path=log_dir_path, console_prefix="off")
    
        log("Passed assert statement: assert add(4, 7) == 11")
        log('Show: calculation_report("add",4,7)')
        log(f'    {calculation_report("add", 4, 7)}')
        
        log_file_path = log.main_log_file_path  # must assign before log.close()
        log.close()
    
        captured = capsys.readouterr()
        console_output = captured.out + captured.err
        
        assert log_file_path is not None
        log_content = log_file_path.read_text(encoding="utf-8")
    
        assert "Calculation report" in console_output
        assert "Calculation report" in log_content
    
        print("")
        print("--- test_01_verify_add_calculation_and_output_to_console_and_log ---")
        print("Passed assert statements that expected strings are in console and log.")
    
        print("\n--- CAPTURED CONSOLE OUTPUT FOR VISUAL INSPECTION ---")
        print(console_output.rstrip())
        print("\n--- LOG CONTENT FOR VISUAL INSPECTION ---")
        print(log_content.rstrip())
       

Example test-file log 
----------------------
Excerpt from `test_calculator.log`:

    test_01_verify_add_calculation_and_output_to_console_and_log
    Passed assert statements: confirmed expected strings in console and log.

    --- CAPTURED CONSOLE OUTPUT FOR VISUAL INSPECTION ---
    Logging started:  2026-08-06 10:03:16
    Passed assert statement: assert add(4, 7) == 11
    ...

    --- LOG CONTENT FOR VISUAL INSPECTION ---
    10:03:16.919 | INFO     | Passed assert statement: assert add(4, 7) == 11
    ...

    ================================ tests coverage ================================
    Name                               Stmts   Miss Branch BrPart   Cover   Missing
    -------------------------------------------------------------------------------
    src/sample_package/calculator.py      22     10      6      2  50.00%   21, 50, 68, 86-96
    -------------------------------------------------------------------------------
    TOTAL                                 24     10      6      2  53.33%
    PASSED tests/test_calculator.py::test_01_verify_add_calculation_and_output_to_console_and_log
    pytest exit code: 0
    duration: 0.42 seconds

- The individual test-file logs include warnings and detailed tracebacks for 
  failed tests if relevant.
- The `TOTAL` coverage of 53.33% applies to this test file only. Other test files 
  boosted total coverage to 76%.
- The `Missing` source lines can help target code that needs additional tests.
- A complete basic project is available in the GitHub repository under

    `examples/basic_project/`.
