Skip to main content
`Ctrl`+`K`

# Tutorial: Simulation via SLURM

Warning

This tutorial is in a draft outline state. It is provided as reference and will be fleshed out in future releases. Be sure to check back in on this tutorial or watch the Changelog for updates!

This tutorial implements the same workflow introduced in Tutorial 04: Simulation, but executes the simulation with the SLURM workload manager.

## References

- SLURM sbatch [73]

## Environment

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

1. Create the tutorials environment if it doesn’t exist

   Linux/MacOS

       $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest

   Windows

       PS > conda create --name waves-tutorial-env --channel conda-forge waves scons matplotlib pandas pyyaml xarray seaborn numpy salib pytest

2. Activate the environment

   Linux/MacOS

       $ conda activate waves-tutorial-env

   Windows

       PS > conda activate waves-tutorial-env

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s `PATH` or by modifying the SConstruct search paths provided to the `waves.scons_extensions.add_program()` method.

## Directory Structure

3. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

Linux/MacOS

    $ mkdir -p ~/waves-tutorials
    $ cd ~/waves-tutorials
    $ pwd
    /home/roppenheimer/waves-tutorials

Windows

    PS > New-Item $HOME\waves-tutorials -ItemType "Directory"
    PS > Set-Location $HOME\waves-tutorials
    PS > Get-Location

    Path
    ----
    C:\Users\roppenheimer\waves-tutorials

Note

If you skipped any of the previous tutorials, run the following commands to create a copy of the necessary tutorial files.

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ waves fetch --overwrite --tutorial 4 && mv tutorial_04_simulation_SConstruct SConstruct
    Destination directory: '/home/roppenheimer/waves-tutorials'

4. Download and copy the `tutorial_04_simulation` file to a new file named `tutorial_sbatch` with the WAVES Command-Line Utility fetch subcommand.

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ waves fetch --overwrite tutorials/tutorial_04_simulation && cp tutorial_04_simulation tutorial_sbatch
    WAVES fetch
    Destination directory: '/home/roppenheimer/waves-tutorials'

## SConscript

A `diff` against the `tutorial_04_simulation` file from Tutorial 04: Simulation is included below to help identify the changes made in this tutorial.

waves-tutorials/tutorial_sbatch

    --- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_04_simulation
    +++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_sbatch
    @@ -56,7 +56,7 @@
     # Comment used in tutorial code snippets: marker-3

     # SolverPrep
    -copy_source_list = [
    +abaqus_source_list = [
         "#/modsim_package/abaqus/rectangle_compression.inp",
         "#/modsim_package/abaqus/assembly.inp",
         "#/modsim_package/abaqus/boundary.inp",
    @@ -65,39 +65,19 @@
         "#/modsim_package/abaqus/parts.inp",
         "#/modsim_package/abaqus/history_output.inp",
     ]
    -workflow.extend(
    -    env.CopySubstfile(
    -        copy_source_list,
    -    )
    -)
    +abaqus_source_list = [pathlib.Path(source_file) for source_file in abaqus_source_list]
    +workflow.extend(env.CopySubstfile(abaqus_source_list))

     # Comment used in tutorial code snippets: marker-4

     # Abaqus Solve
    -solve_source_list = [
    -    "rectangle_compression.inp",
    -    "assembly.inp",
    -    "boundary.inp",
    -    "field_output.inp",
    -    "materials.inp",
    -    "parts.inp",
    -    "history_output.inp",
    -    "rectangle_mesh.inp",
    -]
    -
    +solve_source_list = [source_file.name.rstrip(".in") for source_file in abaqus_source_list]
    +solve_source_list.append("rectangle_mesh.inp")
    +common_suffixes = ("odb", "dat", "msg", "com", "prt")
    +datacheck_suffixes = ("023", "mdl", "sim", "stt")
     datacheck.extend(
         env.AbaqusSolver(
    -        target=[
    -            "rectangle_compression_DATACHECK.odb",
    -            "rectangle_compression_DATACHECK.dat",
    -            "rectangle_compression_DATACHECK.msg",
    -            "rectangle_compression_DATACHECK.com",
    -            "rectangle_compression_DATACHECK.prt",
    -            "rectangle_compression_DATACHECK.023",
    -            "rectangle_compression_DATACHECK.mdl",
    -            "rectangle_compression_DATACHECK.sim",
    -            "rectangle_compression_DATACHECK.stt",
    -        ],
    +        target=[f"rectangle_compression_DATACHECK.{suffix}" for suffix in common_suffixes + datacheck_suffixes],
             source=solve_source_list,
             job="rectangle_compression_DATACHECK",
             program_options="-double both -datacheck",
    @@ -106,16 +86,10 @@

     # Comment used in tutorial code snippets: marker-5

    +abaqus_implicit_extensions = ("sta",)
     workflow.extend(
         env.AbaqusSolver(
    -        target=[
    -            "rectangle_compression.odb",
    -            "rectangle_compression.dat",
    -            "rectangle_compression.msg",
    -            "rectangle_compression.com",
    -            "rectangle_compression.prt",
    -            "rectangle_compression.sta",
    -        ],
    +        target=[f"rectangle_compression.{suffix}" for suffix in common_suffixes + abaqus_implicit_extensions],
             source=solve_source_list,
             job="rectangle_compression",
             program_options="-double both",

Note that the new `AbaqusSolver` builder will be conditionally defined in the `SConstruct` file according to the availability of the `sbatch` command. If `sbatch` is not available, the `slurm_job` variable will go unused by the `waves.scons_extensions.abaqus_solver_builder_factory`()` builder. Since SCons builders don’t throw errors for unused keyword arguments, we do not need to define the task twice in the `SConscript` file.

## SConstruct

A `diff` against the `SConstruct` file from Tutorial 04: Simulation is included below to help identify the changes made in this tutorial.

waves-tutorials/SConstruct

    --- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_04_simulation_SConstruct
    +++ /home/runner/work/waves/waves/build/docs/tutorials_tutorial_sbatch_SConstruct
    @@ -69,6 +69,7 @@
     env["ABAQUS_PROGRAM"] = env.AddProgram(
         env["abaqus_commands"] if env["abaqus_commands"] is not None else default_abaqus_commands
     )
    +env["SBATCH_PROGRAM"] = env.AddProgram(["sbatch"])

     # Comments used in tutorial code snippets: marker-4

    @@ -89,8 +90,21 @@

     # Comments used in tutorial code snippets: marker-5

    +
     # Add builders and pseudo-builders
    -env.Append(BUILDERS={})
    +def solve_builder_select():
    +    """Run the SLURM sbatch builder if available. Fall back to AbaqusSolver builder in not."""
    +    if env["SBATCH_PROGRAM"]:
    +        return waves.scons_extensions.sbatch_abaqus_solver_builder_factory(program="${ABAQUS_PROGRAM}")
    +    else:
    +        return waves.scons_extensions.abaqus_solver_builder_factory(program="${ABAQUS_PROGRAM}")
    +
    +
    +env.Append(
    +    BUILDERS={
    +        "AbaqusSolver": solve_builder_select(),
    +    }
    +)

     # Comments used in tutorial code snippets: marker-6

    @@ -100,6 +114,7 @@
         "tutorial_02_partition_mesh",
         "tutorial_03_solverprep",
         "tutorial_04_simulation",
    +    "tutorial_sbatch",
     ]
     for workflow in workflow_configurations:
         build_dir = env["variant_dir_base"] / workflow

## Build Targets

5. Build the new targets

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ scons tutorial_sbatch

## Output Files

Explore the contents of the `build` directory using the `tree` command against the `build` directory, as shown below. Note that the output files from the previous tutorials also exist in the `build` directory, but the directory is specified by name to reduce clutter in the ouptut shown.

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ tree build/tutorial_sbatch/