Skip to main content
`Ctrl`+`K`

# Tutorial: Writing Builders

This tutorial will introduce WAVES style builders using the `waves.scons_extensions.first_target_builder_factory()` with considerations for handling some representative types of numeric solver behavior. For an example of writing more generic SCons Builders, see the SCons user manual [32]. For a quickstart with simple builders that mimic the WAVES builder directory change operations, see the SCons Quickstart.

## References

- SCons Quickstart

- SCons Builders [32]

- SCons AddPostAction and SCons AddPreAction [32]

## Environment

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

1. Create the tutorials environment if it doesn’t exist

   Linux/MacOS

       $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest

   Windows

       PS > conda create --name waves-tutorial-env --channel conda-forge waves scons matplotlib pandas pyyaml xarray seaborn numpy salib pytest

2. Activate the environment

   Linux/MacOS

       $ conda activate waves-tutorial-env

   Windows

       PS > conda activate waves-tutorial-env

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s `PATH` or by modifying the SConstruct search paths provided to the `waves.scons_extensions.add_program()` method.

Warning

STOP! Before continuing, check that the documentation version matches your installed package version.

1. You can find the documentation version in the upper-left corner of the webpage.

2. You can find the installed WAVES version with `waves --version`.

If they don’t match, you can launch identically matched documentation with the WAVES Command-Line Utility docs subcommand as `waves docs`.

## Directory Structure

3. Create a new `tutorial_writing_builders` directory with the `waves fetch` command below

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ waves fetch --destination tutorial_writing_builders tutorials/tutorial_writing_builders
    $ ls tutorial_writing_builders
    SConstruct implicit.yaml implicit_workflow pyproject.toml scons_extensions.py solver test_scons_extensions.py

4. Make the new `tutorial_writing_builders` directory the current working directory

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ cd tutorial_writing_builders
    $ pwd
    /home/roppenheimer/waves-tutorials/tutorial_writing_builders
    $ ls tutorial_writing_builders
    SConstruct implicit.yaml implicit_workflow pyproject.toml scons_extensions.py solver test_scons_extensions.py

5. (Linux and MacOS only) Make the `solver.py` file executable. In a shell, use the command

   $ chmod +x solver.py

## Solver

Before writing a builder, it is important to understand the solver’s file handling behaviors, exit behaviors, and command line interface options. The specific nature of the numeric algorithms and solution procedures are important considerations when choosing a solver, but when writing a builder only the interfaces with the file system and shell are important. To write a robust builder, you need four things from the solver

1. Deterministic output file paths at workflow configure time

2. Well defined exit criteria

3. Command line options for non-interactive behavior control

4. Non-interactive output file overwrite control options

It is also helpful for the solver to provide

5. Complete control over all output file paths and file base names, including logs

Deterministic output file paths are necessary to allow builder users to define targets for downstream task consumption. While it is possible to write a builder based on `waves.scons_extensions.first_target_emitter()` that does not track any solver owned output files, such a builder is difficult to robustly integrate in a workflow. If solver output file paths are non-deterministic at workflow configure time or difficult to control from the CLI, it is still possible to write a builder by tracking the STDOUT and STDERR messages and redirecting them to a task-specific `*.stdout` file. However, this pushes more effort of workflow configuration onto end users of the builder. If all or some output file paths can be determined at configure time it is possible to write an associated SCons Emitters to automatically populate known builder targets.

Well defined exit criteria is necessary to return reliable task states on solver failures and successes. Most builders should avoid modifying the default solver behavior to provide users with the documented, and therfore expected, behavior. However, when solvers under report non-zero exit codes for fatal errors, false positives, it may be necessary to do extra work, such as log file inspection, to return more reliable exit codes and avoid attempting downstream tasks when the solver task is known to have failed. This is especially important when the solver writes partial output results that may propagate false positives further down the workflow. False negatives can be equally frustrating when the solver results files are sufficiently complete for the user’s workflow and should not be considered fatal. Because desirable task outcome behavior is often project or even workflow specific, it is usually best to maintain the documented solver behavior when writing builders for general distribution.

Command line options for non-interactive behavior control are necessary to provide automatic execution of large workflows and parametric studies. A large benefit of using build systems is providing consistent, reproducible behavior. This is much more difficult when when interactive controls allow workflow behavioral changes. Additionally, interactive controls do not scale well. Finally, when using the `waves.scons_extensions.first_target_builder_factory()` and associated `waves.scons_extensions.first_target_emitter()`, the solver STDOUT and STDERR are redirected to a task-specific log file to provide at least one target file and make task-specific trouble-shooting less cumbersome. It is possible to modify the builder factory’s redirection operation to simultaneously stream to STDOUT/STDERR and a redirected task log file; however, the commands implementing such behavior are not common to all operating systems and shells and is therefore best left to the end user and project specific needs. As a default, a less cluttered STDOUT is often more useful for end users tracking workflow execution, too.

Output overwrite controls are important when writing builders. If the solver increments output file names, workflow re-execution will create file system state dependent targets. Technically these may be deterministic at configure time. However, as a practical matter these are difficult to predict and functionally non-deterministic from the perspective of the builder author because they may depend on the end user’s task construction. If the solver provides an option to overwrite a deterministic output file name, the builder and any associated emitter are much easier to write and test for completeness.

If the output files are controllable and deterministic, then it is possible to work around a missing overwrite feature by adding a pre-solver action that performs file system cleanup as part of the builder. Writing a multi-action builder is outside the scope of the current tutorial, but there are examples in SCons Multi-Action Tasks and the `waves.scons_extensions.action_list_strings()` and `waves.scons_extensions.action_list_scons()` functions can help advanced users pull apart existing builders to augment the action list.

The placeholder solver used in this tutorial does no real work, but provides some common output handling use cases as examples for considerations in builder design. As seen in the solver documentation below, the solver writes a log file with no path controls and no overwrite behavior. The solver builder can not reliably manage this file, except to change to the target parent directory prior to solver execution to ensure that the log is co-located with other task output files. The output file names are deterministic, but require task specific knowledge of the solver options. As a first draft, the builder will make no attempt to predict output file names and instead rely on the user’s knowledge of solver behavior and task options to write complete target lists. There is an overwrite option, which the builder will require to make task behavior more predictable. There is no interactive behavior to work around and also no CLI option to force non-interactive behavior.

Normally the internal implementation and API are not accessible to solver users. Here, the full internal API is produced as an example of Python command line utility and to allow users to play with solver behavior to more closely match solvers of interest. While the Python implementation is written to WAVES style and functional programming best practices, the file handling behavior itself is undesirable when users have direct control over command line utility behavior.

### Solver documentation

Example for common commercial and research solver behavior and handling

Warning

The solver I/O handling and CLI are NOT good behaviors for authors and developers of command line utilities. They are intended to be examples of representative types of challenging behaviors to consider when writing SCons builders.

Solver I/O behaviors:

1. Log files are written to the current working directory. Log files are never overwritten and must be cleaned manually. Log files are automatically incremented from `solver.log` to `solver.log10`. If no free log file number is found and the maximum number of 10 is found, the solver exits with a non-zero exit code.

2. The output file name is built in preferred order: CLI `--output-file` argument or the `--input-file` argument with the replacement extension `.out`.

3. The implicit and explicit routines write output file(s) based on the requested number of threads. If only one thread is requested, the output file name is used. If more than one thread, N, is requested, each thread writes to a separately numbered output file 0 to (N-1) as `output_file.out{number}`.

4. If any output file exists and the overwrite behavior is not requested, the solver exits with a non-zero exit code.

Runtime errors are returned as non-zero exit codes. Internal errors are returned as the appropriate exception.

Exit codes:

1. error loading YAML input file

2. mismatched subcommand and input file routine request

3. output file exists and no overwrite was requested

4. reached max log file integer before finding a free file name

solver.configure(args: Namespace) → dict

  Return the configuration with appended executable information

  Parameters:

    args – The command line argument namespace

  Raises:

    RuntimeError – if the subcommand doesn’t match the input file routine

solver.explicit(args: Namespace) → None

  Explicit routine

  Parameters:

    args – The command line argument namespace

solver.get_parser() → ArgumentParser

  Return the argparse CLI parser

solver.implicit(args: Namespace) → None

  Implicit routine

  Parameters:

    args – The command line argument namespace

solver.main()

  Main function implementing the command line interface and program flow

solver.name_log_file(log_file: Path, max_iterations: int = 10) → Path

  Return the first free log file name

  Parameters:

    - log_file – Log file base name

    - max_iterations – Maximum number of allowable log files

  Raises:

    RuntimeError – if no log file name is free within the max iterations

solver.name_output_file(input_file: Path, output_file: Path) → Path

  Create the output file name from the input file if not specified

solver.positive_nonzero_int(argument)

  Type function for argparse - positive, non-zero integers

  Parameters:

    argument (str) – string argument from argparse

  Returns:

    argument

  Return type:

    int

  Raises:

    ValueError –

    - The argument can’t be cast to int

    - The argument is less than 1

solver.read_input(input_file: Path) → dict

  Return the configuration by reading the input file and handling common errors

  Parameters:

    input_file – The input YAML file absolute or relative path

  Raises:

    RuntimeError – if the YAML file can not be read

solver.solve(configuration: dict) → None

  Common solve logic because we do not really have separate routines

  Parameters:

    configuration – The solver configuration

  Raises:

    RuntimeError – if any output file already exists and overwrite is not requested.

solver.solve_output_files(output_file: Path, solve_cpus: int) → List[Path]

  Return the solve output file list to match the number of solve cpus

  Parameters:

    - output_file – base name for the output file

    - solve_cpus – integer number of solve cpus

## Builder

The tutorial’s builder factory is written as a project specific Python module such that it could be documented and packaged for distribution. It is also possible to write builder factories and builders directly into SCons configuration files.

The builder itself is largely boilerplate documentation and a pass-through to the `waves.scons_extensions.first_target_builder_factory()`. Strictly speaking, the builder factory could be written much more compactly by passing unchanged arguments through as `*kwargs`. Hear, they are kept in the builder factory definition to make documentation of the factory defaults easier and to explicitly set the defaults independently from the WAVES function.

Writing the entire builder from scratch would not require many extra lines of code. The use of the WAVES function is primarily to provide consistency with WAVES builder factories. Builder factory authors are encouraged to read the SCons Builders documentation when their solver needs are not met by the WAVES template builder factories.

     1import waves
     2import SCons.Builder
     3
     4
     5# Write an example custom builder using the WAVES builder factory template
     6def solver_builder_factory(
     7    environment: str = "",
     8    action_prefix: str = "cd ${TARGET.dir.abspath} &&",
     9    program: str = "solver.py",
    10    program_required: str = "",
    11    program_options: str = "",
    12    subcommand: str = "implicit",
    13    subcommand_required: str = "--input-file ${SOURCES[0].abspath} --output-file=${TARGETS[0].abspath} --overwrite",
    14    subcommand_options: str = "",
    15    action_suffix: str = "> ${TARGETS[-1].abspath} 2>&1",
    16    emitter=waves.scons_extensions.first_target_emitter,
    17    **kwargs,
    18) -> SCons.Builder.Builder:
    19    """
    20    This builder factory extends :meth:`waves.scons_extensions.first_target_builder_factory`. This builder factory uses
    21    the :meth:`waves.scons_extensions.first_target_emitter`. At least one task target must be specified in the task
    22    definition and the last target will always be the expected STDOUT and STDERR redirection output file,
    23    ``TARGETS[-1]`` ending in ``*.stdout``.
    24
    25    .. warning::
    26
    27       Users overriding the ``emitter`` keyword argument are responsible for providing an emitter with equivalent STDOUT
    28       file handling behavior as :meth:`waves.scons_extensions.first_target_emitter` or updating the ``action_suffix``
    29       option to match their emitter's behavior.
    30
    31    With the default options this builder requires the following sources file provided in the order:
    32
    33    1. solver program's routine subcommand input file: ``*.yaml``
    34
    35    .. code-block::
    36       :caption: action string construction
    37
    38       ${environment} ${action_prefix} ${program} ${program_required} ${program_options} ${subcommand} ${subcommand_required} ${subcommand_options} ${action_suffix}
    39
    40    .. code-block::
    41       :caption: action string default expansion
    42
    43       ${environment} cd ${TARGET.dir.abspath} && solver.py ${program_required} ${program_optional} implicit --input-file ${SOURCE.abspath} --output-file ${TARGETS[0].abspath} ${subcommand_options} > ${TARGETS[-1].abspath} 2>&1
    44
    45    .. code-block::
    46       :caption: SConstruct
    47
    48       import waves
    49       env = waves.scons_extensions.WAVESEnvironment()
    50       env.AddMethod(waves.scons_extensions.add_program, "AddProgram")
    51       env["solver"] = env.AddProgram(["solver"])
    52       env.Append(BUILDERS={"Solver": solver_explicit_builder_factory()})
    53       env.Solver(
    54           target=["target.out"],
    55           source=["source.yaml"],
    56       )
    57
    58    The builder returned by this factory accepts all SCons Builder arguments. The arguments of this function are also
    59    available as keyword arguments of the builder. When provided during task definition, the task keyword arguments
    60    override the builder keyword arguments.
    61
    62    :param environment: This variable is intended primarily for use with builders and tasks that can not execute from an
    63        SCons construction environment. For instance, when tasks execute on a remote server with SSH wrapped actions
    64        using :meth:`waves.scons_extensions.ssh_builder_actions` and therefore must initialize the remote environment as
    65        part of the builder action.
    66    :param action_prefix: This variable is intended to perform directory change operations prior to program execution
    67    :param program: The solver program absolute or relative path
    68    :param program_required: Space delimited string of required solver program options and arguments that are crucial to
    69        builder behavior and should not be modified except by advanced users
    70    :param program_options: Space delimited string of optional solver program options and arguments that can be freely
    71        modified by the user
    72    :param subcommand: The solver program's routine subcommand absolute or relative path
    73    :param subcommand_required: Space delimited string of required solver program's routine subcommand options and
    74        arguments that are crucial to builder behavior and should not be modified except by advanced users.
    75    :param subcommand_options: Space delimited string of optional solver program's routine subcommand options and
    76        arguments that can be freely modified by the user
    77    :param action_suffix: This variable is intended to perform program STDOUT and STDERR redirection operations.
    78    :param emitter: An SCons emitter function. This is not a keyword argument in the action string.
    79    :param kwargs: Any additional keyword arguments are passed directly to the SCons builder object.
    80
    81    :return: Solver journal builder
    82    """  # noqa: E501
    83    builder = waves.scons_extensions.first_target_builder_factory(
    84        environment=environment,
    85        action_prefix=action_prefix,
    86        program=program,
    87        program_required=program_required,
    88        program_options=program_options,
    89        subcommand=subcommand,
    90        subcommand_required=subcommand_required,
    91        subcommand_options=subcommand_options,
    92        action_suffix=action_suffix,
    93        emitter=emitter,
    94        **kwargs,
    95    )
    96    return builder

Example unit tests to verify the builder action string, keywords, emitter, and node count are also provided. These are similar to the WAVES unit test verification suite for builder facotories based on `waves.scons_extensions.first_target_builder_factory()` and should be the minimum verification implemented for similar builders.

Unit tests are not sufficient to verify desired solver behavior. System tests that run a real, but minimal, example problem should also be implemented to ensure that the builder action string, task re-execution, file handling expecations, and command line options are appropriate for workflow use. The system tests should be complete enough to verify that the third-party solver behavior still matches the assumptions in the buider design. For WAVES, the tutorials serve as system tests for builder design. For a project specific builder, a minimal working example should be included in the project as an inexpensive verification check and easy-to-use troubleshooting task.

      1import waves
      2import pytest
      3import SCons.Environment
      4
      5import scons_extensions
      6
      7
      8def dummy_emitter_for_testing(target, source, env):
      9    return target, source
     10
     11
     12solver_builder_factory_tests = {
     13    "default behavior": ({}, {}, ["solver_builder_factory.out0"], False, 2, 1),
     14    "different emitter": ({}, {}, ["solver_builder_factory.out1"], dummy_emitter_for_testing, 1, 1),
     15    "builder kwargs overrides": (
     16        {
     17            "environment": "different environment",
     18            "action_prefix": "different action prefix",
     19            "program": "different program",
     20            "program_required": "different program required",
     21            "program_options": "different program options",
     22            "subcommand": "different subcommand",
     23            "subcommand_required": "different subcommand required",
     24            "subcommand_options": "different subcommand options",
     25            "action_suffix": "different action suffix",
     26        },
     27        {},
     28        ["solver_builder_factory.out2"],
     29        False,
     30        2,
     31        1,
     32    ),
     33    "task kwargs overrides": (
     34        {},
     35        {
     36            "environment": "different environment",
     37            "action_prefix": "different action prefix",
     38            "program": "different program",
     39            "program_required": "different program required",
     40            "program_options": "different program options",
     41            "subcommand": "different subcommand",
     42            "subcommand_required": "different subcommand required",
     43            "subcommand_options": "different subcommand options",
     44            "action_suffix": "different action suffix",
     45        },
     46        ["solver_builder_factory.out3"],
     47        False,
     48        2,
     49        1,
     50    ),
     51}
     52
     53
     54# TODO: Expose WAVES builder factory test functions for end users
     55@pytest.mark.parametrize(
     56    "builder_kwargs, task_kwargs, target, emitter, expected_node_count, expected_action_count",
     57    solver_builder_factory_tests.values(),
     58    ids=solver_builder_factory_tests.keys(),
     59)
     60def test_solver_builder_factory(
     61    builder_kwargs: dict,
     62    task_kwargs: dict,
     63    target: list,
     64    emitter,
     65    expected_node_count: int,
     66    expected_action_count: int,
     67) -> None:
     68    """Template test for builder factories based on :meth:`waves.scons_extensions.builder_factory`
     69
     70    :param builder_kwargs: Keyword arguments unpacked at the builder instantiation
     71    :param task_kwargs: Keyword arguments unpacked at the task instantiation
     72    :param target: Explicit list of targets provided at the task instantiation
     73    :param emitter: A custom factory emitter. Mostly intended as a pass-through check. Set to ``False`` to avoid
     74        providing an emitter argument to the builder factory.
     75    :param expected_node_count: The expected number of target nodes.
     76    :param expected_action_count: The expected number of target node actions.
     77    """
     78    # Set default expectations to match default argument values
     79    expected_kwargs = {
     80        "environment": "",
     81        "action_prefix": "cd ${TARGET.dir.abspath} &&",
     82        "program": "solver.py",
     83        "program_required": "",
     84        "program_options": "",
     85        "subcommand": "implicit",
     86        "subcommand_required": "--input-file ${SOURCES[0].abspath} --output-file=${TARGETS[0].abspath} --overwrite",
     87        "subcommand_options": "",
     88        "action_suffix": "> ${TARGETS[-1].abspath} 2>&1",
     89    }
     90
     91    # Update expected arguments to match test case
     92    expected_kwargs.update(builder_kwargs)
     93    expected_kwargs.update(task_kwargs)
     94    # Expected action matches the pre-SCons-substitution string with newline delimiter
     95    expected_action = (
     96        "${environment} ${action_prefix} ${program} ${program_required} ${program_options} "
     97        "${subcommand} ${subcommand_required} ${subcommand_options} ${action_suffix}"
     98    )
     99
    100    # Handle additional builder kwargs without changing default behavior
    101    expected_emitter = waves.scons_extensions.first_target_emitter
    102    emitter_handling = {}
    103    if emitter is not False:
    104        expected_emitter = emitter
    105        emitter_handling.update({"emitter": emitter})
    106
    107    # Test builder object attributes
    108    factory = getattr(scons_extensions, "solver_builder_factory")
    109    builder = factory(**builder_kwargs, **emitter_handling)
    110    assert builder.action.cmd_list == expected_action
    111    assert builder.emitter == expected_emitter
    112
    113    # Assemble the builder and a task to interrogate
    114    env = SCons.Environment.Environment()
    115    env.Append(BUILDERS={"Builder": builder})
    116    nodes = env.Builder(
    117        target=target,
    118        source=["check_builder_factory.in"],
    119        **task_kwargs,
    120    )
    121
    122    # Test task definition node counts, action(s), and task keyword arguments
    123    assert len(nodes) == expected_node_count
    124    for node in nodes:
    125        node.get_executor()
    126        assert len(node.executor.action_list) == expected_action_count
    127        assert str(node.executor.action_list[0]) == expected_action
    128    for node in nodes:
    129        for key, expected_value in expected_kwargs.items():
    130            assert node.env[key] == expected_value

## SConscript

The task target list depends on the solver options. The builder is not written to manage or inspect the solver command line options and does not include an emitter to match the builder target expectations to the solver command line options. Instead, the end user is expected to manage the target list directly.

As in many of the tutorials, the number of solve cpus is accepted from the project specific command line arguments. The SConscript file implementing an implicit solver workflow inspects the construction environment solve cpus option prior to configuring the task to determine the expected target list. Since the solver overrides the output file extension, no special handling of the `--output-file` option is required to truncate or correct the provided extension when the target list changes.

Since the solver does not provide any overwrite behavior for the log file, a user owned clean options is provided to clean the entire workflow build directory in addition to the known target list. This is necessary to help workflow users purge log files and to avoid solver task failures when the maximum number of log files is reached. More advanced builder authors might choose to implement a log file clean action prior to the solver action. More advanced end users might prefer to use an SCons AddPreAction to perform log file cleaning even when the builder does not. Without either implementation, users must remember to clean their build directory regularly to avoid solver exits from accumulated log files.

     1#! /usr/bin/env python
     2import pathlib
     3
     4Import("env")
     5
     6build_directory = pathlib.Path(Dir(".").abspath)
     7workflow_name = build_directory.name
     8
     9output_file = "implicit.out"
    10if env["solve_cpus"] > 1:
    11    target = [f"{output_file}{number}" for number in range(env["solve_cpus"])]
    12else:
    13    target = [output_file]
    14workflow = env.Solver(
    15    target=target,
    16    source=["implicit.yaml"],
    17    solve_cpus=env["solve_cpus"],
    18)
    19env.Clean(workflow, [Dir(build_directory)])
    20env.Alias(workflow_name, workflow)

## SConstruct

The SConstruct file of this tutorial is a stripped down example of the core tutorials. All features of the project configuration should look familiar to the core tutorials. It implements a variant build directory, solve cpus control, and environment recovery from the launching shell. Since the solver behaves like a project specific file, the solver executable is found by absolute path from the tutorial/project directory and the parent directory is added to the environment `PATH`. Besides the default action construction, the project adds a project-specific solver option and the `solve_cpus` task keyword argument.

     1#! /usr/bin/env python
     2import os
     3import pathlib
     4
     5import waves
     6
     7import scons_extensions
     8
     9
    10AddOption(
    11    "--build-dir",
    12    dest="variant_dir_base",
    13    default="build",
    14    nargs=1,
    15    type="string",
    16    action="store",
    17    metavar="DIR",
    18    help="SCons build (variant) root directory. Relative or absolute path. (default: '%default')",
    19)
    20AddOption(
    21    "--solve-cpus",
    22    dest="solve_cpus",
    23    default=1,
    24    nargs=1,
    25    type="int",
    26    action="store",
    27    metavar="N",
    28    help="Run the solver task using N CPUs. (default: '%default')",
    29)
    30
    31env = waves.scons_extensions.WAVESEnvironment(
    32    ENV=os.environ.copy(),
    33    variant_dir_base=pathlib.Path(GetOption("variant_dir_base")),
    34    solve_cpus=GetOption("solve_cpus"),
    35)
    36env["ENV"]["PYTHONDONTWRITEBYTECODE"] = 1
    37
    38# Print failed task *.stdout files
    39env.PrintBuildFailures()
    40
    41# Empty defaults list to avoid building all simulation targets by default
    42env.Default()
    43
    44# Add project builders and scanners
    45solver_executable = pathlib.Path("solver.py").resolve()
    46env["solver"] = env.AddProgram([solver_executable])
    47
    48# Users should typically rely on AddProgram behavior for third-party software as above, which checks for execute
    49# permissions and PATH resolution. If a tutorial user forgets to change the execute permissions on Linux, or is running
    50# on Windows, force execution with the Python interpretter and absolute path.
    51#
    52# VVV Code specific to making tutorial execution more robust VVV
    53if env["solver"] is None:
    54    env["solver"] = f"python {solver_executable}"
    55# ^^^ Code specific to making tutorial execution more robust ^^^
    56
    57env.Append(
    58    BUILDERS={
    59        "Solver": scons_extensions.solver_builder_factory(
    60            program=env["solver"], subcommand_options="$(--solve-cpus=${solve_cpus}$)"
    61        ),
    62    }
    63)
    64
    65# Add simulation targets
    66workflow_configurations = [
    67    "pytest",
    68    "implicit_workflow",
    69]
    70for workflow in workflow_configurations:
    71    build_dir = env["variant_dir_base"] / workflow
    72    SConscript(workflow, variant_dir=build_dir, exports={"env": env}, duplicate=False)
    73
    74# Print project local help
    75env.ProjectHelp()

## Building targets

First, run the workflow with the default number of solve cpus and observe the output directory.

    $ scons implicit_workflow
    scons: Reading SConscript files ...
    Checking whether /home/roppenheimer/waves-tutorials/tutorial_writing_builders/solver.py program exists.../home/roppenheimer/waves-tutorials/tutorial_writing_builders/solver.py
    scons: done reading SConscript files.
    scons: Building targets ...
    cd /home/roppenheimer/waves-tutorials/tutorial_writing_builders/build/implicit_workflow && solver.py implicit --input-file /home/roppenheimer/waves-tutorials/tutorial_writing_builders/implicit.yaml --output-file=/home/roppenheimer/waves-tutorials/tutorial_writing_builders/build/implicit_workflow/implicit.out --overwrite --solve-cpus=1 > /home/roppenheimer/waves-tutorials/tutorial_writing_builders/build/implicit_workflow/implicit.out.stdout 2>&1
    scons: done building targets.
    $ find build -type f
    build/implicit_workflow/implicit.out
    build/implicit_workflow/implicit.out.stdout
    build/implicit_workflow/solver.log

Re-run the workflow with two solve cpus. The task re-runs because the target list has changed. With the target list change, the original, single cpu solve output file is left around and not overwritten. A more advanced builder or user task definition is required to clean up previous files when the expected target file list changes. If the solver or builder factory provided an option to merge the multiple cpu output files, it could be possible for the user to write a task that does not re-execute when the number of solve cpus changes. In this tutorial, the user would need to write the merge operation as an SCons AddPostAction to get similar behavior when the requested solver cpus changed.

Observe that the previous log file still exists and a new log file with extension `*.log1` is found in the build output. Cleaning log files to produce deterministic output for task log file tracking and cleaning would require a more advanced user task definition. It would be tempting to write a builder action that always purges log files prior to execution; however, if the end user were running multiple solver tasks in this build directory, all log files of previous tasks would also be removed. In this case, it is probably better to leave log file handling to the end user.

Finally, the user target list construction combined with the builder `--output-file` option handling results in the `*.stdout` file name change from `implicit.out.stdout` to `implicit.out0.stdout`. A builder factory emitter could be written to look for the `solve_cpus` environment varible to match emitted solver output targets to solver output handling. Changing the `*.stdout` naming convention for greater consistency is possible to handle in the builder factory, but would be more difficult to implement robustly and document clearly, so matching the extension of the first expected target is probably best.

    $ scons implicit_workflow --solve-cpus=2
    scons: Reading SConscript files ...
    Checking whether /home/roppenheimer/waves-tutorials/tutorial_writing_builders/solver.py program exists.../home/roppenheimer/waves-tutorials/tutorial_writing_builders/solver.py
    scons: done reading SConscript files.
    scons: Building targets ...
    cd /home/roppenheimer/waves-tutorials/tutorial_writing_builders/build/implicit_workflow && solver.py implicit --input-file /home/roppenheimer/waves-tutorials/tutorial_writing_builders/implicit.yaml --output-file=/home/roppenheimer/waves-tutorials/tutorial_writing_builders/build/implicit_workflow/implicit.out0 --overwrite --solve-cpus=2 > /home/roppenheimer/waves-tutorials/tutorial_writing_builders/build/implicit_workflow/implicit.out0.stdout 2>&1
    scons: done building targets.
    $ find build -type f
    build/implicit_workflow/implicit.out0
    build/implicit_workflow/implicit.out
    build/implicit_workflow/solver.log1
    build/implicit_workflow/implicit.out1
    build/implicit_workflow/implicit.out.stdout
    build/implicit_workflow/implicit.out0.stdout
    build/implicit_workflow/solver.log

Run the clean operation with the two solve cpus option and observe that the entire builder directory is removed. This is necessary for the end user to have consistent log file purging behavior without resorting to shell remove commands. Removing build directory artifacts by shell command is not necessarily a bad practices, but can build up muscle memory for commands that are unnecessarily destructive, such as removing the entire build directory with `rm -r build`. In a single workflow project like this tutorial this achieves the same result. But in a project with many or computationally expensive workflows this muscle memory may result in expensive data loss and `scons workflow --clean` operations should be preferred, but not relied upon, for better data retention habits.

    $ scons implicit_workflow --solve-cpus=2 --clean
    scons: Reading SConscript files ...
    Checking whether /home/roppenheimer/waves-tutorials/tutorial_writing_builders/solver.py program exists.../home/roppenheimer/waves-tutorials/tutorial_writing_builders/solver.py
    scons: done reading SConscript files.
    scons: Cleaning targets ...
    Removed build/implicit_workflow/implicit.out0
    Removed build/implicit_workflow/implicit.out1
    Removed build/implicit_workflow/implicit.out0.stdout
    Removed build/implicit_workflow/implicit.out
    Removed build/implicit_workflow/implicit.out.stdout
    Removed build/implicit_workflow/solver.log
    Removed build/implicit_workflow/solver.log1
    Removed directory build/implicit_workflow
    scons: done cleaning targets.
    $ find build -type f
    $