Skip to main content
`Ctrl`+`K`

# Tutorial 11: Regression Testing

Regression testing is the practice of running a verification test suite after making changes to a repository or codebase. For modsim repositories, there may not be many unit or integration tests if there is no software or scripting library specific to the project. Instead, regression testing a modsim repository may look more like regular execution of system tests that verify the simulation workflow still executes as expected.

Ideally, this verification suite of system tests would perform the complete simulation workflow from start to finish. However, modsim repositories often contain simulations that are computationally expensive or produce large amounts of data on disk. In these cases, it may be too expensive to run the full simulation suite at any regular interval. It is still desirable to provide early warning of breaking changes in the simulation workflow, so as much of the workflow that can be tested should be tested as regularly as possible given compute resource constraints.

This tutorial introduces a project wide alias to allow convenient execution of the simulation workflow through the simulation datacheck task introduced in Tutorial 04: Simulation. From that tutorial onward, each tutorial has propagated a tutorial specific datacheck alias. This tutorial will add a project wide `datacheck` alias and apply it to a copy of the Tutorial 09: Post-Processing configuration files. The user may also go back to previous tutorials to include the full suite of datacheck tasks in the project wide datacheck regression test alias.

In addition to the datachecks, this tutorial will introduce a full simulation results regression test script and task. The regression test task will be added to the regular workflow alias to run everytime the full workflow is run. This test compares the actual simulation results within a float tolerance. Comprehensive results regression testing is valuable to evaluate changes to important quantities of interest when software versions change, e.g. when installing a new version of Abaqus.

After this tutorial, the workflow will have three sets of tests: fast running unit tests introduced in Tutorial 10: Unit Testing, relatively fast running simulation preparation checks with the `datacheck` alias, and full simulation results regression tests. The tutorial simulations run fast enough that performing the full suite of tests for every change in the project is tractable. However, in practice, projects may choose to run only the unit tests and datachecks on a per-change basis and reserve the comprehensive results testing for a scheduled regression suite.

## References

- Regression Testing

- SCons Alias [32]

- Continuous Integration software

  - GitHub Actions: https://docs.github.com/en/actions [57]

  - Gitlab CI: https://docs.gitlab.com/ee/ci/ [58]

  - Bitbucket Pipelines: https://bitbucket.org/product/features/pipelines [59]

## Environment

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

1. Create the tutorials environment if it doesn’t exist

   Linux/MacOS

       $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest

   Windows

       PS > conda create --name waves-tutorial-env --channel conda-forge waves scons matplotlib pandas pyyaml xarray seaborn numpy salib pytest

2. Activate the environment

   Linux/MacOS

       $ conda activate waves-tutorial-env

   Windows

       PS > conda activate waves-tutorial-env

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s `PATH` or by modifying the SConstruct search paths provided to the `waves.scons_extensions.add_program()` method.

Warning

STOP! Before continuing, check that the documentation version matches your installed package version.

1. You can find the documentation version in the upper-left corner of the webpage.

2. You can find the installed WAVES version with `waves --version`.

If they don’t match, you can launch identically matched documentation with the WAVES Command-Line Utility docs subcommand as `waves docs`.

## Directory Structure

3. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

Linux/MacOS

    $ mkdir -p ~/waves-tutorials
    $ cd ~/waves-tutorials
    $ pwd
    /home/roppenheimer/waves-tutorials

Windows

    PS > New-Item $HOME\waves-tutorials -ItemType "Directory"
    PS > Set-Location $HOME\waves-tutorials
    PS > Get-Location

    Path
    ----
    C:\Users\roppenheimer\waves-tutorials

Note

If you skipped any of the previous tutorials, run the following commands to create a copy of the necessary tutorial files.

Linux/MacOS

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ waves fetch --overwrite --tutorial 10 && mv tutorial_10_unit_testing_SConstruct SConstruct
    WAVES fetch
    Destination directory: '/home/roppenheimer/waves-tutorials'

Windows

    PS > Get-Location

    Path
    ----
    C:\Users\roppenheimer\waves-tutorials

    PS > waves fetch --overwrite --tutorial 10 && Move-Item tutorial_10_unit_testing_SConstruct SConstruct -Force
    WAVES fetch
    Destination directory: 'C:\Users\roppenheimer\waves-tutorials'

4. Download and copy the `tutorial_09_post_processing` file to a new file named `tutorial_11_regression_testing` with the WAVES Command-Line Utility fetch subcommand.

Linux/MacOS

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ waves fetch --overwrite tutorials/tutorial_09_post_processing && cp tutorial_09_post_processing tutorial_11_regression_testing
    WAVES fetch
    Destination directory: '/home/roppenheimer/waves-tutorials'

Windows

    PS > Get-Location

    Path
    ----
    C:\Users\roppenheimer\waves-tutorials

    PS > waves fetch --overwrite tutorials\tutorial_09_post_processing && Copy-Item tutorial_09_post_processing tutorial_11_regression_testing
    WAVES fetch
    Destination directory: 'C:\Users\roppenheimer\waves-tutorials'

## Regression Script

5. In the `waves-tutorials/modsim_package/python` directory, review the file named `regression.py` introduced in Tutorial 10: Unit Testing.

waves-tutorials/modsim_package/python/regression.py

      1#!/usr/bin/env python
      2import sys
      3import pathlib
      4import argparse
      5
      6import pandas
      7import yaml
      8
      9
     10def sort_dataframe(dataframe, index_column="time", sort_columns=["time", "set_name"]):
     11    """Return a sorted dataframe and set an index
     12
     13    1. sort columns by column name
     14    2. sort rows by column values ``sort_columns``
     15    3. set an index
     16
     17    :returns: sorted and indexed dataframe
     18    :rtype: pandas.DataFrame
     19    """
     20    return dataframe.reindex(sorted(dataframe.columns), axis=1).sort_values(sort_columns).set_index(index_column)
     21
     22
     23def csv_files_match(current_csv, expected_csv, index_column="time", sort_columns=["time", "set_name"]):
     24    """Compare two pandas DataFrame objects and determine if they match.
     25
     26    :param pandas.DataFrame current_csv: Current CSV data of generated plot.
     27    :param pandas.DataFrame expected_csv: Expected CSV data.
     28
     29    :returns: True if the CSV files match, False otherwise.
     30    :rtype: bool
     31    """
     32    current = sort_dataframe(current_csv, index_column=index_column, sort_columns=sort_columns)
     33    expected = sort_dataframe(expected_csv, index_column=index_column, sort_columns=sort_columns)
     34    try:
     35        pandas.testing.assert_frame_equal(current, expected)
     36    except AssertionError as err:
     37        print(
     38            f"The CSV regression test failed. Data in expected CSV file and current CSV file do not match.\n{err}",
     39            file=sys.stderr,
     40        )
     41        equal = False
     42    else:
     43        equal = True
     44    return equal
     45
     46
     47def main(
     48    first_file: pathlib.Path,
     49    second_file: pathlib.Path,
     50    output_file: pathlib.Path,
     51) -> None:
     52    """Compare CSV files and return an error code if they differ
     53
     54    :param first_file: path-like or file-like object containing the first CSV dataset
     55    :param second_file: path-like or file-like object containing the second CSV dataset
     56    """
     57    regression_results = {}
     58
     59    # CSV regression file comparison
     60    first_data = pandas.read_csv(first_file)
     61    second_data = pandas.read_csv(second_file)
     62    regression_results.update({"CSV comparison": csv_files_match(first_data, second_data)})
     63
     64    with open(output_file, "w") as output:
     65        output.write(yaml.safe_dump(regression_results))
     66
     67    if len(regression_results.values()) < 1 or not all(regression_results.values()):
     68        sys.exit("One or more regression tests failed")
     69
     70
     71def get_parser() -> argparse.ArgumentParser:
     72    """Return parser for CLI options
     73
     74    All options should use the double-hyphen ``--option VALUE`` syntax to avoid clashes with the Abaqus option syntax,
     75    including flag style arguments ``--flag``. Single hyphen ``-f`` flag syntax often clashes with the Abaqus command
     76    line options and should be avoided.
     77
     78    :returns: parser
     79    :rtype:
     80    """
     81    script_name = pathlib.Path(__file__)
     82    default_output_file = f"{script_name.stem}.yaml"
     83
     84    prog = f"python {script_name.name} "
     85    cli_description = "Compare CSV files and return an error code if they differ"
     86    parser = argparse.ArgumentParser(description=cli_description, prog=prog)
     87    parser.add_argument(
     88        "FIRST_FILE",
     89        type=pathlib.Path,
     90        help="First CSV file for comparison",
     91    )
     92    parser.add_argument(
     93        "SECOND_FILE",
     94        type=pathlib.Path,
     95        help="Second CSV file for comparison",
     96    )
     97    parser.add_argument(
     98        "--output-file",
     99        type=pathlib.Path,
    100        default=default_output_file,
    101        help="Regression test pass/fail list",
    102    )
    103    return parser
    104
    105
    106if __name__ == "__main__":
    107    parser = get_parser()
    108    args = parser.parse_args()
    109    main(
    110        args.FIRST_FILE,
    111        args.SECOND_FILE,
    112        args.output_file,
    113    )

## CSV file

6. In the `waves-tutorials/modsim_package/python` directory, create a new file named `rectangle_compression_cartesian_product.csv` from the contents below

waves-tutorials/modsim_package/python/rectangle_compression_cartesian_product.csv

     1time,set_name,step,elements,integrationPoint,E values,E,S values,S,width,height,global_seed,displacement,set_hash
     20.0175000000745058,parameter_set0,Step-1,1,1.0,E22,-0.000159091,S22,-0.0159091,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f
     30.0175000000745058,parameter_set1,Step-1,1,1.0,E22,-0.000159091,S22,-0.0159091,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
     40.0175000000745058,parameter_set2,Step-1,1,1.0,E22,-0.000175,S22,-0.0175,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
     50.0175000000745058,parameter_set3,Step-1,1,1.0,E22,-0.000175,S22,-0.0175,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7
     60.0709374994039536,parameter_set0,Step-1,1,1.0,E22,-0.000644886,S22,-0.0644886,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f
     70.0709374994039536,parameter_set1,Step-1,1,1.0,E22,-0.000644886,S22,-0.0644886,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
     80.0709374994039536,parameter_set2,Step-1,1,1.0,E22,-0.000709375,S22,-0.0709375,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
     90.0709374994039536,parameter_set3,Step-1,1,1.0,E22,-0.000709375,S22,-0.0709375,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7
    100.251289069652557,parameter_set0,Step-1,1,1.0,E22,-0.00228445,S22,-0.228445,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f
    110.251289069652557,parameter_set1,Step-1,1,1.0,E22,-0.00228445,S22,-0.228445,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
    120.251289069652557,parameter_set2,Step-1,1,1.0,E22,-0.00251289,S22,-0.251289,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
    130.251289069652557,parameter_set3,Step-1,1,1.0,E22,-0.00251289,S22,-0.251289,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7
    140.859975576400757,parameter_set0,Step-1,1,1.0,E22,-0.00781796,S22,-0.781796,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f
    150.859975576400757,parameter_set1,Step-1,1,1.0,E22,-0.00781796,S22,-0.781796,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
    160.859975576400757,parameter_set2,Step-1,1,1.0,E22,-0.00859976,S22,-0.859976,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
    170.859975576400757,parameter_set3,Step-1,1,1.0,E22,-0.00859976,S22,-0.859976,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7
    181.0,parameter_set0,Step-1,1,1.0,E22,-0.00909091,S22,-0.909091,1.1,1.1,1.0,-0.01,181cb00b478ced26819416aeef6a1a3f
    191.0,parameter_set1,Step-1,1,1.0,E22,-0.00909091,S22,-0.909091,1.0,1.1,1.0,-0.01,53980146e2729b8956ecf6ef39f342b4
    201.0,parameter_set2,Step-1,1,1.0,E22,-0.01,S22,-1.0,1.0,1.0,1.0,-0.01,cf0934b22f43400165bd3d34aa61013f
    211.0,parameter_set3,Step-1,1,1.0,E22,-0.01,S22,-1.0,1.1,1.0,1.0,-0.01,fff52e9be95e50cc872ec321280d91a7

This file represents a copy of previous simulation results that the project has stored as the reviewed and approved simulation results. The regression task will compare these “past” results with the current simulation results produced during the post-processing task introduced in Tutorial 09: Post-Processing `rectangle_compression_cartesian_product.csv` using the regression.py CLI.

## SConscript

A `diff` against the `tutorial_09_post_processing` file from Tutorial 09: Post-Processing is included below to help identify the changes made in this tutorial.

waves-tutorials/tutorial_11_regression_testing

    --- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_09_post_processing
    +++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_11_regression_testing
    @@ -5,6 +5,8 @@

     * ``env`` - The SCons construction environment with the following required keys

    +  * ``datacheck_alias`` - String for the alias collecting the datacheck workflow targets
    +  * ``regression_alias`` - String for the alias collecting the regression test suite targets
       * ``unconditional_build`` - Boolean flag to force building of conditionally ignored targets
       * ``abaqus`` - String path for the Abaqus executable
     """
    @@ -176,11 +178,27 @@
         )
     )

    +# Regression test
    +workflow.extend(
    +    env.PythonScript(
    +        target=["regression.yaml"],
    +        source=[
    +            "#/modsim_package/python/regression.py",
    +            "stress_strain_comparison.csv",
    +            "#/modsim_package/python/rectangle_compression_cartesian_product.csv",
    +        ],
    +        subcommand_options="${SOURCES[1:].abspath} --output-file ${TARGET.abspath}",
    +    )
    +)
    +
     # Collector alias based on parent directory name
     env.Alias(workflow_name, workflow)
     env.Alias(f"{workflow_name}_datacheck", datacheck)
    +env.Alias(env["datacheck_alias"], datacheck)
    +env.Alias(env["regression_alias"], datacheck)

     if not env["unconditional_build"] and not env["ABAQUS_PROGRAM"]:
         print(f"Program 'abaqus' was not found in construction environment. Ignoring '{workflow_name}' target(s)")
         Ignore([".", workflow_name], workflow)
         Ignore([".", f"{workflow_name}_datacheck"], datacheck)
    +    Ignore([".", env["datacheck_alias"], env["regression_alias"]], datacheck)

There are two changes made in this tutorial. The first is to compare the expected simulation results to the current simulation’s output. A new task compares the expected results as the CSV file created above against the current simulation output with the new Python script, regression.py. See the regression.py CLI documentation for a description of the post-processing script’s behavior.

The second change adds a dedicated alias for the datacheck targets to allow partial workflow execution. This is useful when a full simulation may take a long time, but the simulation preparation is worth testing on a regular basis. We’ve also added the regression alias introduced briefly in Tutorial 10: Unit Testing. Previously, this alias was a duplicate of the `unit_testing` workflow alias. Now this alias can be used as a collector alias for running a regression suite with a single command, while preserving the ability to run the unit tests as a standalone workflow.

Here we add the datacheck targets as an example of running a partial workflow as part of the regression test suite. For fast running simulations, it would be valuable to run the full simulation and post-processing with CSV results testing as part of the regular regression suite. For large projects with long running simulations, several regression aliases may be added to facilitate testing at different intervals. For instance, the datachecks might be run everytime the project changes, but the simulations might be run on a weekly schedule with a `regression_weekly` alias that includes the full simulations in addition to the unit tests and datachecks.

## SConstruct

A `diff` against the `SConstruct` file from Tutorial 10: Unit Testing is included below to help identify the changes made in this tutorial.

waves-tutorials/SConstruct

    --- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_10_unit_testing_SConstruct
    +++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_11_regression_testing_SConstruct
    @@ -82,6 +82,7 @@
         "project_dir": project_dir,
         "version": version,
         "regression_alias": "regression",
    +    "datacheck_alias": "datacheck",
     }
     for key, value in project_variables.items():
         env[key] = value
    @@ -112,6 +113,7 @@
         "tutorial_07_cartesian_product",
         "tutorial_08_data_extraction",
         "tutorial_09_post_processing",
    +    "tutorial_11_regression_testing",
     ]
     for workflow in workflow_configurations:
         build_dir = env["variant_dir_base"] / workflow

## Build Targets

7. Build the datacheck targets without executing the full simulation workflow

Linux/MacOS

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ time scons datacheck --jobs=4
    <output truncated>
    scons: done building targets.

    real 0m9.952s
    user 0m21.537s
    sys  0m15.664s

Windows

    PS > Get-Location

    Path
    ----
    C:\Users\roppenheimer\waves-tutorials

    PS > Measure-Command { scons datacheck --jobs=4 | Out-Default }
    <output truncated>
    scons: done building targets.

    Days              : 0
    Hours             : 0
    Minutes           : 0
    Seconds           : 10
    Milliseconds      : 129
    Ticks             : 101291696
    TotalDays         : 0.0001172357592593
    TotalHours        : 0.002813658222222
    TotalMinutes      : 0.1688194933333
    TotalSeconds      : 10.1291696
    TotalMilliseconds : 10129.1696

8. Run the full workflow and verify that the CSV regression test passes

Linux/MacOS

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ scons datacheck --clean
    $ time scons tutorial_11_regression_testing --jobs=4
    <output truncated>
    scons: done building targets.

    real 0m29.031s
    user 0m25.712s
    sys  0m25.622s
    $ cat build/tutorial_11_regression_testing/regression.yaml
    CSV comparison: true

Windows

    PS > Get-Location

    Path
    ----
    C:\Users\roppenheimer\waves-tutorials

    PS > scons datacheck --clean
    PS > Measure-Command { scons tutorial_11_regression_testing --jobs=4 | Out-Default }
    <output truncated>
    scons: done building targets.

    Days              : 0
    Hours             : 0
    Minutes           : 0
    Seconds           : 31
    Milliseconds      : 493
    Ticks             : 314936692
    TotalDays         : 0.0003645100601852
    TotalHours        : 0.008748241444444
    TotalMinutes      : 0.5248944866667
    TotalSeconds      : 31.4936692
    TotalMilliseconds : 31493.6692

    PS > Get-Content build\tutorial_11_regression_testing\regression.yaml
    CSV comparison: true

If you haven’t added the project-wide datacheck alias to the previous tutorials, you should expect the `datacheck` alias to run faster than the `tutorial_11_regression_testing` alias because the datacheck excludes the solve, extract, and post-processing tasks. In these tutorials, the difference in execution time is not large. However, in many production modsim projects, the simulations may require hours or even days to complete. In that case, the relatively fast running solverprep verification may be tractable for regular testing where the full simulations and post-processing are not.

To approximate the time savings of the new project-wide `datacheck` alias for a (slightly) larger modsim project, you can go back through the previous tutorials and add each tutorial’s datacheck task to the new alias. For a fair comparison, you will also need to add a comparable alias to collect the full workflow for each tutorial, e.g. `full_workflows`. You can then repeat the `time` commands above with a more comprehensive `datacheck` and `full_workflows` aliases.

## Output Files

Linux/MacOS

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ tree build/tutorial_11_regression_testing/parameter_set0/
    build/tutorial_11_regression_testing/parameter_set0/
    |-- abaqus.rpy
    |-- abaqus.rpy.1
    |-- abaqus.rpy.2
    |-- assembly.inp
    |-- boundary.inp
    |-- field_output.inp
    |-- history_output.inp
    |-- materials.inp
    |-- parts.inp
    |-- rectangle_compression.inp
    |-- rectangle_compression.inp.in
    |-- rectangle_compression_DATACHECK.023
    |-- rectangle_compression_DATACHECK.com
    |-- rectangle_compression_DATACHECK.dat
    |-- rectangle_compression_DATACHECK.mdl
    |-- rectangle_compression_DATACHECK.msg
    |-- rectangle_compression_DATACHECK.odb
    |-- rectangle_compression_DATACHECK.prt
    |-- rectangle_compression_DATACHECK.sim
    |-- rectangle_compression_DATACHECK.stdout
    |-- rectangle_compression_DATACHECK.stt
    |-- rectangle_geometry.cae
    |-- rectangle_geometry.jnl
    |-- rectangle_geometry.stdout
    |-- rectangle_mesh.cae
    |-- rectangle_mesh.inp
    |-- rectangle_mesh.jnl
    |-- rectangle_mesh.stdout
    |-- rectangle_partition.cae
    |-- rectangle_partition.jnl
    `-- rectangle_partition.stdout

    0 directories, 31 files
    $ tree build/tutorial_11_regression_testing/ -L 1
    build/tutorial_11_regression_testing/
    |-- parameter_set0
    |-- parameter_set1
    |-- parameter_set2
    |-- parameter_set3
    |-- parameter_study.h5
    |-- regression.yaml
    |-- regression.yaml.stdout
    |-- stress_strain_comparison.csv
    |-- stress_strain_comparison.pdf
    `-- stress_strain_comparison.pdf.stdout

    4 directories, 6 files

Windows

    PS > Get-Location

    Path
    ----
    C:\Users\roppenheimer\waves-tutorials

    PS > tree build\tutorial_11_regression_testing\parameter_set0\ /F
    C:\USERS\ROPPENHEIMER\WAVES-TUTORIALS\BUILD\TUTORIAL_11_REGRESSION_TESTING\PARAMETER_SET0
        abaqus.rpy
        abaqus.rpy.2
        abaqus.rpy.3
        abaqus.rpy.4
        abaqus.rpy.5
        assembly.inp
        boundary.inp
        field_output.inp
        history_output.inp
        materials.inp
        parts.inp
        rectangle_compression.com
        rectangle_compression.csv
        rectangle_compression.dat
        rectangle_compression.h5
        rectangle_compression.inp
        rectangle_compression.inp.in
        rectangle_compression.msg
        rectangle_compression.odb
        rectangle_compression.odb.stdout
        rectangle_compression.prt
        rectangle_compression.sta
        rectangle_compression_DATACHECK.cax
        rectangle_compression_datasets.h5
        rectangle_geometry.cae
        rectangle_geometry.cae.stdout
        rectangle_geometry.jnl
        rectangle_mesh.cae
        rectangle_mesh.inp
        rectangle_mesh.inp.stdout
        rectangle_mesh.jnl
        rectangle_partition.cae
        rectangle_partition.cae.stdout
        rectangle_partition.jnl

    No subfolders exist

    PS > Get-ChildItem build\tutorial_11_regression_testing\

        Directory: C:\Users\roppenheimer\waves-tutorials\build\tutorial_11_regression_testing

    Mode                 LastWriteTime         Length Name
    ----                 -------------         ------ ----
    d----            6/9/2023  4:32 PM                parameter_set0
    d----            6/9/2023  4:32 PM                parameter_set1
    d----            6/9/2023  4:32 PM                parameter_set2
    d----            6/9/2023  4:32 PM                parameter_set3
    -a---            6/9/2023  4:32 PM           9942 parameter_study.h5
    -a---            6/9/2023  4:32 PM             22 regression.yaml
    -a---            6/9/2023  4:32 PM              0 regression.yaml.stdout
    -a---            6/9/2023  4:32 PM           2609 stress_strain_comparison.csv
    -a---            6/9/2023  4:32 PM          12061 stress_strain_comparison.pdf
    -a---            6/9/2023  4:32 PM           1160 stress_strain_comparison.pdf.stdout

## Workflow Visualization

View the workflow directed graph by running the following command and opening the image in your preferred image viewer. Plot the workflow with only the first set, `set0`.

Linux/MacOS

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ waves visualize datacheck --output-file tutorial_11_datacheck_set0.png --width=42 --height=8 --exclude-list /usr/bin .stdout .jnl .prt .com .msg .dat .sta --exclude-regex "set[1-9]"

Windows

    PS > Get-Location

    Path
    ----
    C:\Users\roppenheimer\waves-tutorials

    PS > waves visualize datacheck --output-file tutorial_11_datacheck_set0.png --width=42 --height=8 --exclude-list .stdout .jnl .prt .com .msg .dat .sta --exclude-regex "set[1-9]"

The output should look similar to the figure below.

This tutorial’s `datacheck` directed graph should look different from the graph in Tutorial 09: Post-Processing. Here we have plotted the `datacheck` alias output, which does not execute the full simulation workflow. This partial directed graph may run faster than the full simulation workflow for frequent regression tests.

## Automation

There are many tools that can help automate the execution of the modsim project regression tests. With the collector alias, those tools need only execute a single SCons command to perform the selected, lower cost tasks for simulation workflow verification, `scons datacheck`. If git [21] is used as the version control system, developer operations software such as GitHub [24], Gitlab [25], and Atlassian’s Bitbucket [26] provide continuous integration software that can automate verification tests on triggers, such as merge requests, or on a regular schedule.