Skip to main content
`Ctrl`+`K`

# Tutorial: Quinoa

Warning

This tutorial is in a draft outline state. It is provided as reference and will be fleshed out in future releases. Be sure to check back in on this tutorial or watch the Changelog for updates!

Warning

Most WAVES tutorials are used as system tests in the regression test suite to ensure that the tutorial files are up-to-date and functional. While the Quinoa team refactors around a breaking change in the input file parser, the Quiona tutorial is not part of the regression suite, but the Quinoa builder unit tests are part of the verification suite. If you run into problems running this tutorial, please contact the Quiona development team for the current status of the input file parser.

Warning

Most WAVES tutorials are used as system tests in the regression test suite to ensure that the tutorial files are up-to-date and functional. The `quinoa-local` alias without sbatch submission is part of the regression suite, but neither the sbatch submission nor the `quinoa-remote` alias is regression tested. The SSH remote excution and sbatch behavior was tested when the tutorial was written and is expected to run correctly. If you run into problems running this tutorial, please contact the WAVES development team.

## References

- Cubit: Python Interface [62]

- Cubit: Importing Cubit into Python [62]

- Quinoa: Quinoa documentation [74, 75]

- Charm++ [76]

## Environment

SCons and WAVES can be installed in a Conda environment with the Conda package manager. See the Conda installation and Conda environment management documentation for more details about using Conda.

Note

The SALib and numpy versions may not need to be this strict for most tutorials. However, Tutorial: Sensitivity Study uncovered some undocumented SALib version sensitivity to numpy surrounding the numpy v2 rollout.

1. Create the tutorials environment if it doesn’t exist

   Linux/MacOS

       $ conda create --name waves-tutorial-env --channel conda-forge waves 'scons>=4.6' matplotlib pandas pyyaml xarray seaborn 'numpy>=2' 'salib>=1.5.1' pytest

   Windows

       PS > conda create --name waves-tutorial-env --channel conda-forge waves scons matplotlib pandas pyyaml xarray seaborn numpy salib pytest

2. Activate the environment

   Linux/MacOS

       $ conda activate waves-tutorial-env

   Windows

       PS > conda activate waves-tutorial-env

Some tutorials require additional third-party software that is not available for the Conda package manager. This software must be installed separately and either made available to SConstruct by modifying your system’s `PATH` or by modifying the SConstruct search paths provided to the `waves.scons_extensions.add_program()` method.

## Directory Structure

3. Create and change to a new project root directory to house the tutorial files if you have not already done so. For example

Linux/MacOS

    $ mkdir -p ~/waves-tutorials
    $ cd ~/waves-tutorials
    $ pwd
    /home/roppenheimer/waves-tutorials

Windows

    PS > New-Item $HOME\waves-tutorials -ItemType "Directory"
    PS > Set-Location $HOME\waves-tutorials
    PS > Get-Location

    Path
    ----
    C:\Users\roppenheimer\waves-tutorials

4. Create a new `tutorial_quinoa` directory with the `waves fetch` command below

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ waves fetch --destination tutorial_quinoa tutorials/tutorial_quinoa
    $ ls tutorial_quinoa
    box.py flow.lua SConstruct SConscript

5. Make the new `tutorial_quinoa` directory the current working directory

    $ pwd
    /home/roppenheimer/waves-tutorials
    $ cd tutorial_quinoa
    $ pwd
    /home/roppenheimer/waves-tutorials/tutorial_quinoa
    $ ls
    box.py flow.lua SConstruct SConscript

## SConscript

6. Review the `SConscript` file.

The structure has changed enough from the core tutorials that a diff view is not as useful. Instead the contents of the new SConscript files are duplicated below.

waves-tutorials/tutorial_quinoa/SConscript

     1#! /usr/bin/env python
     2Import("env", "envQuinoa")
     3
     4# TODO: move parameters to a dedicated dictionary file read
     5simulation_variables = {
     6    "xlength": 1.0,
     7    "ylength": 0.5,
     8    "zlength": 0.5,
     9}
    10
    11workflow = []
    12remote_directory_prefix = env["remote_directory"]
    13
    14# Geometry/partition/mesh
    15# TODO: separate the local and remote workflows into dedicated SConscript files that re-use the cubit workflow as in the
    16# Cubit+Abaqus and Cubit+Sierra tutorials
    17mesh = env.PythonScript(
    18    target=["box.cub", "box.exo"],
    19    source=["box.py"],
    20    subcommand_options="--output-file ${TARGET.abspath} --xlength ${xlength} --ylength ${ylength} --zlength ${zlength}",
    21    **simulation_variables,
    22)
    23workflow.extend(mesh)
    24env.Alias("mesh", mesh)
    25
    26# SSH remote build on HPC
    27remote_directory = remote_directory_prefix / "remote"
    28remote_target = env.SSHQuinoaSolver(
    29    target=["remote.stdout"],
    30    source=["flow.lua", "box.exo"],
    31    charmrun="charmrun",
    32    inciter="inciter",
    33    remote_directory=remote_directory,
    34)
    35env.Alias("quinoa-remote", remote_target)
    36
    37# Local build on local server
    38local_target = envQuinoa.QuinoaSolver(
    39    target=["local.stdout"],
    40    source=["flow.lua", "box.exo"],
    41    charmrun=envQuinoa["charmrun"],
    42    inciter=envQuinoa["inciter"],
    43)
    44envQuinoa.Alias("quinoa-local", local_target)
    45
    46if not env["unconditional_build"] and not envQuinoa["inciter"]:
    47    print("Program 'inciter' was not found in construction environment. Ignoring 'quinoa-local' target(s)")
    48    Ignore([".", "quinoa-local"], local_target)

## Cubit Journal Files

7. Review the `box.py` Cubit journal file

The Cubit journal file includes a similar CLI introduced in Tutorial 02: Partition and Mesh for the Abaqus journal files. Besides the differences in Abaqus and Cubit commands, the major difference between the Abaqus and Cubit journal files is the opportunity to use Python 3 with Cubit, where Abaqus journal files must use the Abaqus controlled installation of Python 2. Unlike the core tutorials, this journal file only parameterizes the geometry, so intermediate partition and mesh journal files are not as useful and the journal file performs all operations at once.

waves-tutorials/tutorial_quinoa/box.py

     1import argparse
     2import pathlib
     3
     4import cubit
     5
     6
     7def main(output_file, xlength, ylength, zlength):
     8    """Write brick geometry with specified dimensions
     9
    10    :param str output_file: The output file for the Cubit model. Will be stripped of the extension and ``.cub`` and
    11        ``.exo`` will be used for the model and mesh files, respectively.
    12    :param xlength: box edge length on global x axis
    13    :param ylength: box edge length on global y axis
    14    :param zlength: box edge length on global z axis
    15
    16    :returns: writes ``output_file``.cub and ``output_file``.exo
    17    """
    18    cubit.init(["cubit", "-noecho", "-nojournal", "-nographics", "-batch"])
    19    cubit.cmd("new")
    20    cubit.cmd("reset")
    21
    22    cubit_file = pathlib.Path(output_file).with_suffix(".cub")
    23    exodus_file = cubit_file.with_suffix(".exo")
    24
    25    # Geometry
    26    cubit.cmd(f"brick x {xlength} y {ylength} z {zlength}")
    27    cubit.cmd(f"move volume 1 x {xlength * 0.5} y {ylength * 0.5} z {zlength * 0.5}")
    28    cubit.cmd(f"webcut volume 1 with plane xplane offset {xlength * 0.5} noimprint nomerge")
    29    cubit.cmd("merge volume 1 2")
    30
    31    # Sets (partition)
    32    cubit.cmd("sideset 1 add surface 8 15")
    33    cubit.cmd("sideset 2 add surface 9 16")
    34    cubit.cmd("sideset 3 add surface 10 14")
    35    cubit.cmd("sideset 4 add surface 4")
    36    cubit.cmd("sideset 5 add surface 11 13")
    37    cubit.cmd("sideset 6 add surface 6")
    38
    39    # Mesh
    40    cubit.cmd("Trimesher surface gradation 1.05")
    41    cubit.cmd("Trimesher volume gradation 1.05")
    42
    43    cubit.cmd("surface 6 7 8 9 10 11 13 14 15 16 Scheme TriMesh geometry approximation angle 10")
    44    cubit.cmd("surface 13 14 15 16  size 0.065")
    45    cubit.cmd("surface 4 size 0.09")
    46    cubit.cmd("surface 7 size .09")
    47    cubit.cmd("surface 8 9 10 11 size 0.45")
    48    cubit.cmd("surface 6 size 0.09")
    49
    50    cubit.cmd("mesh surface 6 7 8 9 10 11 13 14 15 16")
    51
    52    cubit.cmd("volume 1 2 Scheme Tetmesh proximity layers off geometry approximation angle 10")
    53    cubit.cmd("volume 1 2 Tetmesh growth_factor 1.001")
    54    cubit.cmd("volume 1 size 0.065")
    55    cubit.cmd("volume 2 size 0.065")
    56
    57    cubit.cmd("mesh volume 1 2")
    58    cubit.cmd("block 1 add volume 1 2")
    59
    60    cubit.cmd("set exodus netcdf4 off")
    61    cubit.cmd("set large exodus file on")
    62    cubit.cmd(f"export Genesis '{exodus_file}' dimension 3 block 1  overwrite")
    63    cubit.cmd(f"save as '{cubit_file}' overwrite")
    64
    65
    66def get_parser():
    67    """Return the command line parser"""
    68    script_name = pathlib.Path(__file__)
    69    prog = f"python {script_name.name} "
    70    cli_description = "Write brick geometry ``output_file``.cub and ``output_file``.exo with specified dimensions"
    71    parser = argparse.ArgumentParser(description=cli_description, prog=prog)
    72    parser.add_argument(
    73        "--output-file",
    74        type=str,
    75        required=True,
    76        # fmt: off
    77        help="The output file for the Cubit model. Will be stripped of the extension and ``.cub`` and ``.exo`` will "
    78             "be used for the model and mesh files, respectively.",
    79        # fmt: on
    80    )
    81    parser.add_argument("--xlength", type=float, required=True, help="box edge length on global x axis")
    82    parser.add_argument("--ylength", type=float, required=True, help="box edge length on global y axis")
    83    parser.add_argument("--zlength", type=float, required=True, help="box edge length on global z axis")
    84    return parser
    85
    86
    87if __name__ == "__main__":
    88    parser = get_parser()
    89    args = parser.parse_args()
    90    main(
    91        output_file=args.output_file,
    92        xlength=args.xlength,
    93        ylength=args.ylength,
    94        zlength=args.zlength,
    95    )

## Quinoa Input File(s)

8. Create or review the Quinoa input file from the contents below

waves-tutorials/tutorial_quinoa/flow.lua

     1inciter = {
     2
     3  title = "Flow in a simple box" ,
     4
     5  term = 0.11,  -- Max physical time, s
     6  ttyi= 10,  -- TTY output interval
     7  cfl =0.8,
     8  scheme = "alecg",
     9
    10  partitioning = "rcb",
    11
    12  compflow = {
    13    physics ="euler",
    14  },
    15    -- Units: m,kg,s,
    16  ic = {
    17    density = 0.912,  -- kg/m^3
    18    velocity={ 0.0, 0.0, 0.0 },  -- m/s
    19    pressure = 77816.5,  -- Initiate linear
    20  },
    21
    22  material = {
    23    {
    24      gamma = { 1.4 },
    25      cv ={ 717.5 },
    26    }
    27  },
    28
    29  bc = {
    30    {
    31      farfield = {6},  -- Outlet
    32      pressure = 77816.5,
    33      density = 0.912,
    34      velocity = {0.0, 0.0, 0.0},
    35      timedep = {
    36	{
    37          sideset = {4}, --Inlet
    38	  fn = {  -- t  p  rho  xvel yvel zvel
    39            0.0, 77816.5, 0.912, 0.000, 0.000, 0.000,
    40            0.1, 77816.5, 0.912, 5.000, 0.000, 0.000,
    41           10.0, 77816.5, 0.912, 5.000, 0.000, 0.000,
    42	  }
    43	}
    44      },
    45      symmetry = {1,2,3,5},
    46    }
    47  },
    48
    49
    50  history_output = {
    51    interval = 1,
    52    point = {
    53      {id = "p1", coord = { 0.5, 0.25, 0.25 }},
    54    }
    55  },
    56
    57
    58  field_output = {
    59    time_interval = 0.001,
    60    nodevar = {
    61      "density",
    62      "x-velocity",
    63      "y-velocity",
    64      "z-velocity",
    65      "specific_total_energy",
    66      "pressure"
    67    }
    68  },
    69
    70
    71  diagnostics = {
    72    interval= 250,
    73    format  = "scientific",
    74    error   =  "l2"
    75  }
    76
    77}

## SConstruct

Note that Quinoa requires a separate construction environment from the launching Conda environment. This is because the Quinoa compiler/linker and openmpi library conflict with the launching Conda environment. You may need to update the Quinoa activation shell command according to the instructions on your local system.

A `diff` against the `SConstruct` file from Tutorial 04: Simulation is included below to help identify the changes made in this tutorial.

waves-tutorials/tutorial_quinoa/SConstruct

    --- /home/runner/work/waves/waves/build/docs/tutorials_tutorial_04_simulation_SConstruct
    +++ /home/runner/work/waves/waves/build/docs/tutorial_quinoa_SConstruct
    @@ -1,6 +1,7 @@
     #! /usr/bin/env python

     import os
    +import getpass
     import pathlib

     import waves
    @@ -33,20 +34,19 @@
         help="Print task *.stdout target file(s) on build failures. (default: '%default')",
     )
     # Python optparse appends to the default list instead of overriding. Must implement default/override ourselves.
    -default_abaqus_commands = [
    -    "/apps/abaqus/Commands/abq2024",
    -    "/usr/projects/ea/abaqus/Commands/abq2024",
    -    "abq2024",
    -    "abaqus",
    +default_cubit_commands = [
    +    "/apps/Cubit-16.12/cubit",
    +    "/usr/projects/ea/Cubit/Cubit-16.12/cubit",
    +    "cubit",
     ]
     AddOption(
    -    "--abaqus-command",
    -    dest="abaqus_command",
    +    "--cubit-command",
    +    dest="cubit_command",
         nargs=1,
         type="string",
         action="append",
         metavar="COMMAND",
    -    help=f"Override for the Abaqus command. Repeat to specify more than one (default: {default_abaqus_commands})",
    +    help=f"Override for the Cubit command. Repeat to specify more than one (default: {default_cubit_commands})",
     )

     # Comments used in tutorial code snippets: marker-2
    @@ -57,18 +57,31 @@
         variant_dir_base=pathlib.Path(GetOption("variant_dir_base")),
         unconditional_build=GetOption("unconditional_build"),
         print_build_failures=GetOption("print_build_failures"),
    -    abaqus_commands=GetOption("abaqus_command"),
    +    cubit_commands=GetOption("cubit_command"),
     )

     # Conditionally print failed task *.stdout files
     env.PrintBuildFailures(print_stdout=env["print_build_failures"])

    +# Always copy (no sym-links) when duplicating
    +env.SetOption("duplicate", "copy")
    +
     # Comments used in tutorial code snippets: marker-3

     # Find required programs for conditional target ignoring and absolute path for use in target actions
    -env["ABAQUS_PROGRAM"] = env.AddProgram(
    -    env["abaqus_commands"] if env["abaqus_commands"] is not None else default_abaqus_commands
    +env["CUBIT_PROGRAM"] = env.AddCubit(
    +    env["cubit_commands"] if env["cubit_commands"] is not None else default_cubit_commands
     )
    +env["SBATCH_PROGRAM"] = env.AddProgram(["sbatch"])
    +env["user"] = getpass.getuser()
    +
    +# Try a preference ordered list of project server-specific environment modulefiles
    +quinoa_hpc_modulefile = pathlib.Path("/usr/projects/ea/Quinoa/modules/modulefiles/quinoa")
    +quinoa_modulefile_options = [
    +    pathlib.Path("/projects/aea_compute/modulefiles/aea-quinoa"),
    +    quinoa_hpc_modulefile,
    +]
    +quinoa_modulefile = next((path for path in quinoa_modulefile_options if path.exists()), None)

     # Comments used in tutorial code snippets: marker-4

    @@ -80,30 +93,53 @@
         "project_name": project_name,
         "project_dir": project_dir,
         "version": version,
    +    "remote_directory": pathlib.Path(f"/users/{env['user']}/WAVES-TUTORIAL/tutorial_quinoa"),
     }
     for key, value in project_variables.items():
         env[key] = value

    -# Make the project package importable for Python and Abaqus Python environments
    -env.PrependENVPath("PYTHONPATH", project_dir)
    -
     # Comments used in tutorial code snippets: marker-5

     # Add builders and pseudo-builders
    -env.Append(BUILDERS={})
    +env.Append(
    +    BUILDERS={
    +        "SSHQuinoaSolver": waves.scons_extensions.ssh_builder_actions(
    +            waves.scons_extensions.sbatch_quinoa_builder_factory(
    +                environment=f"module use {quinoa_hpc_modulefile.parent} && module load {quinoa_hpc_modulefile.name} &&"
    +            ),
    +            remote_server="ro-rfe.lanl.gov",
    +        )
    +    }
    +)
    +
    +# Source the Quinoa environment and store it separately from the Conda/Cubit environment to avoid PATH interference
    +if quinoa_modulefile:
    +    envQuinoa = waves.scons_extensions.shell_environment(
    +        "module unload aea-nightly aea-weekly aea-quarterly aea-release && "
    +        f"module use {quinoa_modulefile.parent} && "
    +        f"module load {quinoa_modulefile.name}"
    +    )
    +else:
    +    envQuinoa = Environment()
    +envQuinoa.AddMethod(waves.scons_extensions.add_program, "AddProgram")
    +envQuinoa["inciter"] = envQuinoa.AddProgram(["inciter"])
    +envQuinoa["charmrun"] = envQuinoa.AddProgram(["charmrun"])
    +
    +# Add WAVES builders. If sbatch is found, use it.
    +# Should allow the same scons alias to run directly on sstbigbird, but submit as an sbatch job on HPC
    +quinoa_builder_factory = waves.scons_extensions.quinoa_builder_factory
    +if env["SBATCH_PROGRAM"] is not None:
    +    quinoa_builder_factory = waves.scons_extensions.sbatch_quinoa_builder_factory
    +envQuinoa.Append(
    +    BUILDERS={"QuinoaSolver": quinoa_builder_factory(program=envQuinoa["charmrun"], subcommand=envQuinoa["inciter"])}
    +)

     # Comments used in tutorial code snippets: marker-6

     # Add simulation targets
    -workflow_configurations = [
    -    "tutorial_01_geometry",
    -    "tutorial_02_partition_mesh",
    -    "tutorial_03_solverprep",
    -    "tutorial_04_simulation",
    -]
    -for workflow in workflow_configurations:
    -    build_dir = env["variant_dir_base"] / workflow
    -    SConscript(workflow, variant_dir=build_dir, exports={"env": env}, duplicate=False)
    +SConscript(
    +    "SConscript", variant_dir=env["variant_dir_base"], exports={"env": env, "envQuinoa": envQuinoa}, duplicate=True
    +)

     # Comments used in tutorial code snippets: marker-7

Note that the Cubit Python file doesn’t perform any imports from the current modsim project package, so the `PYTHONPATH` modification is no longer required. This tutorial is created in a new, stand-alone subdirectory, so the previous tutorial workflow configurations are no longer available. Only the `quinoa-local` and `quinoa-remote` workflow configurations will be found by SCons at execution time.

## Build Targets

9. Build the new targets

    $ pwd
    /home/roppenheimer/waves-tutorials/tutorial_quinoa
    scons: Reading SConscript files ...
    Checking whether '/apps/Cubit-16.12/cubit' program exists.../apps/Cubit-16.12/cubit
    Checking whether 'cubit' program exists...no
    Checking whether 'sbatch' program exists...no
    Sourcing the shell environment with command 'module use /projects/aea_compute/modulefiles && module load aea-quinoa' ...
    Checking whether 'inciter' program exists.../projects/aea_compute/aea-quinoa/bin/inciter
    Checking whether '/users/cclong/QUINOA/quinoa/buildOS/Main/inciter' program exists...no
    Checking whether 'charmrun' program exists.../projects/aea_compute/aea-quinoa/bin/charmrun
    Checking whether '/users/cclong/QUINOA/quinoa/buildOS/Main/charmrun' program exists...no
    scons: done reading SConscript files.
    scons: Building targets ...
    cd /home/roppenheimer/waves-tutorials/tutorial_quinoa/build && python /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/box.py --output-file /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/box.cub --xlength 1.0 --ylength 0.5 --zlength 0.5 > /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/box.cub.stdout 2>&1
    cd /home/roppenheimer/waves-tutorials/tutorial_quinoa/build && /projects/aea_compute/aea-quinoa/bin/charmrun +p1 /projects/aea_compute/aea-quinoa/bin/inciter --control /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/flow.lua --input /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/box.exo > /home/roppenheimer/waves-tutorials/tutorial_quinoa/build/local.stdout 2>&1
    scons: done building targets.

## Output Files

Explore the contents of the `build` directory using the `tree` command against the `build` directory, as shown below.

    $ pwd
    /home/roppenheimer/waves-tutorials/tutorial_quinoa
    $ tree build/
    build
    |-- SConscript
    |-- box.cub
    |-- box.cub.stdout
    |-- box.exo
    |-- box.py
    |-- diag
    |-- flow.lua
    |-- flow.q
    |-- inciter_input.log
    |-- inciter_screen.log
    |-- local.stdout
    |-- mesh_edge_pdf.0.txt
    |-- mesh_ntet_pdf.0.txt
    |-- mesh_vol_pdf.0.txt
    |-- out.e-s.0.1.0
    `-- restart
        |-- MainChares.dat
        |-- RO.dat
        `-- sub0
            |-- Chares_0.dat
            |-- Groups_0.dat
            |-- NodeGroups_0.dat
            `-- arr_0.dat

    2 directories, 21 files