Skip to main content

OpenFOAM

The OpenFOAM workflow step can be used to start both serial and parallel instances of OpenFOAM commands.

OpenFOAM

Case Directory: The location of your OpenFOAM case. This is the directory which contains your system, constant and 0 directories.

Version: The version of OpenFOAM you would like to use.

Precision: Single or Double Precision of OpenFOAM

Number of CPU cores: The number of cpu cores to use for the OpenFOAM command. Note that, serial only commands must have the Number of CPU cores set to 1.

OpenFOAM Command: The OpenFOAM command you want to run with any additional arguments. Note that, you do not need to include the -parallel option when invoking a parallel solver, this is automatically added for you.

info

Not all versions of OpenFOAM may be available by default on your HPCBOX cluster. Please contact HPCBOX support for assistance with compiling, installing and configuring the binaries. If you plan on compiling and installing yourself, please follow the installation process outlined in installation

Running multiple instances​

You can launch multiple instances of an OpenFOAM case to work on different input files. These runs are started as a single job and a directory pattern can be provided to start each instance in the case directory associated with it.

OpenFOAM Multiple instances

Start: The start number to use as a suffix to the directory pattern.

End: The end number to use as a suffix to the directory pattern.

Increment: The increment to add for each instance suffix.

Instance Log: Checking this option places a log file for each run separately within the run directory. This is in addition to the consolidated log file in the job’s main working directory.

Directory naming pattern: The naming pattern to use for each instance run. Each run is started in a subdirectory suffixed with this pattern and the counter generated from the previous options.

Example: For a setting with Start=10, End=100, Increment=10, Directory naming pattern=run_

Each instance of the OpenFOAM run will be started under subdirectories run_10, run_20, run_30, run_40, run_50, run_60, run_70, run_80, run_90 and run_100. You need to make sure your case files are available within these sub directories of the main working directory provided for the job.

Editing Files​

Visual Studio code is available to help you edit your OpenFOAM files and run simple commands through its integrated terminal. A launcher icon for Code can be found on your desktop. To edit files within the OpenFOAM environment, please run the following commands in the integrated terminal.

For OpenFOAM version XXXX
module load openfoam/XXXX
source $OPENFOAM_HOME/OpenFOAM-$OPENFOAM_VERSION/etc/bashrc

OpenFOAM Editing Files

Using OpenFOAM Tutorials​

OpenFOAM tutorial files generally ship with an Allrun file which runs the various steps involved in a successful execution of an OpenFOAM case. A sample Allrun file looks like the following example:

cd ${0%/*} || exit 1    # Run from this directory
# Source tutorial run functions
. $WM_PROJECT_DIR/bin/tools/RunFunctions
# copy DTC hull surface from resources folder
cp $FOAM_TUTORIALS/resources/geometry/DTC-scaled.stl.gz constant/triSurface/
runApplication surfaceFeatures
runApplication blockMesh

for i in 1 2 3 4 5 6
do
    runApplication -s $i \
        topoSet -dict system/topoSetDict.${i}

    runApplication -s $i \
        refineMesh -dict system/refineMeshDict -overwrite
done

runApplication snappyHexMesh -overwrite
runApplication setFields
runApplication decomposePar
runParallel renumberMesh -overwrite
runParallel $(getApplication)
runApplication reconstructPar

Note the use of runParallel and runApplication in the above example file. These shell aliases will not work directly within the HPCBOX application because they are not fine-tuned for every distributed cloud environment. If you want to use one of the tutorials, it is possible to either manually create a workflow step for every command described in the Allrun file, or, use the Allrun file for all the serial operations by commenting out the lines until the first occurrence of runParallel and use HPCBOX for rest of the steps. You must not include runParallel or runApplication when setting up the workflow within HPCBOX.

Setup combining Allrun and HPCBOX​

As an example, the previous Allrun sample file with the commented out parallel steps would look like:

cd ${0%/*} || exit 1    # Run from this directory

# Source tutorial run functions
. $WM_PROJECT_DIR/bin/tools/RunFunctions

# copy DTC hull surface from resources folder
cp $FOAM_TUTORIALS/resources/geometry/DTC-scaled.stl.gz constant/triSurface/

runApplication surfaceFeatures
runApplication blockMesh
for i in 1 2 3 4 5 6
Do
    runApplication -s $i \
        topoSet -dict system/topoSetDict.${i}


    runApplication -s $i \
        refineMesh -dict system/refineMeshDict -overwrite
Done

runApplication snappyHexMesh -overwrite
runApplication setFields
runApplication decomposePar

# The following steps will be set up with HPCBOX

# runParallel renumberMesh -overwrite 
# runParallel $(getApplication)
# runApplication reconstructPar

The modified Allrun shown above can be executed directly within the terminal after setting up the OpenFOAM environment using module files as described in the Editing Files section. Once the case setup is complete, the last three steps, including the runParallel steps can be executed as an HPCBOX workflow.

Equivalent to # runParallel renumberMesh -overwrite

OpenFOAM tutorial step-1

Equivalent to # runParallel $(getApplication)

OpenFOAM tutorial step-2

Equivalent to # runApplication reconstructPar

OpenFOAM tutorial step-2

Paraview​

The Paraview workflow step can be used to launch Paraview for Post-Processing of the OpenFOAM case.

tip

Make sure you check the GPU checkbox if your HPCBOX cluster has a GPU worker for hardware accelerated rendering.

Installation​

If compiling and installing OpenFOAM yourself, may sure it gets compiled and installed under /opt/drz/apps/openfoam/VERSION where VERSION is the OpenFOAM version. For example, v1906

$ ls /opt/drz/apps/openfoam/
v1906  v1906-openmpi

ls /opt/drz/apps/openfoam/v1906
OpenFOAM-v1906  OpenFOAM-v1906.tgz  ThirdParty-v1906  ThirdParty-v1906.tgz

Once installed, place a module file for the specific version of OpenFOAM that was compiled and installed. The file should have the same name as the OpenFOAM version and must be placed under /opt/drz/modulefiles/openfoam. Its contents would be something like the below if using Intel MPI :

#%Module 1.0
#
#  OpenFOAM v1906 module file with Intel MPI OneAPI
#



setenv  OPENFOAM_HOME /opt/drz/apps/openfoam/v1906
setenv  OPENFOAM_VERSION v1906

setenv   DRZ_OPENFOAM_MPI_TYPE intelmpi

set      INTEL_MPI_HOME  /opt/intel/oneapi/mpi/latest

setenv   DRZ_OPENFOAM_IMPI_INTEL_MPI_HOME $INTEL_MPI_HOME
setenv   DRZ_OPENFOAM_IMPI_I_MPI_FABRICS  shm:ofi
setenv   DRZ_OPENFOAM_IMPI_PATH  $INTEL_MPI_HOME/bin
setenv   DRZ_OPENFOAM_IMPI_FI_PROVIDER_PATH $INTEL_MPI_HOME/libfabric/lib/prov
setenv   DRZ_OPENFOAM_IMPI_LD_LIBRARY_PATH $INTEL_MPI_HOME/lib:$INTEL_MPI_HOME/libfabric/lib
setenv   FI_PROVIDER  mlx

# # Set Intel Compiler runtime path
setenv INTEL_COMPILER_HOME /opt/drz/apps/intel/impi/2017



#%Module 1.0
#
#  OpenFOAM v1906 module file with older Intel MPI
#


setenv  OPENFOAM_HOME /opt/drz/apps/openfoam/v1906
setenv  OPENFOAM_VERSION v1906


setenv DRZ_OPENFOAM_MPI_TYPE intelmpi

#setenv   DRZ_OPENFOAM_IMPI_INTEL_MPI_HOME /opt/intel/oneapi/mpi/latest
set      INTEL_MPI_HOME  /opt/intel/impi/2019.5.281
setenv   DRZ_OPENFOAM_IMPI_INTEL_MPI_HOME $INTEL_MPI_HOME
setenv   DRZ_OPENFOAM_IMPI_I_MPI_FABRICS  shm:ofi
#setenv   FI_PROVIDER mlx
#setenv   DRZ_OPENFOAM_IMPI_PATH  /opt/intel/oneapi/mpi/latest/bin
setenv   DRZ_OPENFOAM_IMPI_PATH  $INTEL_MPI_HOME/bin64
setenv   DRZ_OPENFOAM_IMPI_FI_PROVIDER_PATH $INTEL_MPI_HOME/intel64/libfabric/lib/prov
setenv   DRZ_OPENFOAM_IMPI_LD_LIBRARY_PATH $INTEL_MPI_HOME/lib64:$INTEL_MPI_HOME/lib64/release:$INTEL_MPI_HOME/intel64/libfabric/lib/:$INTEL_MPI_HOME/intel64/libfabric/lib/prov

# # Set Intel Compiler runtime path
setenv INTEL_COMPILER_HOME /opt/drz/apps/intel/impi/2017


info

Change the value of OPENFOAM_HOME OPENFOAM_VERSION to match the version you just compiled and installed. Additionally, set Intel MPI specific variables to match the compiler and Intel MPI you used when configuring the OpenFOAM build.

If using OpenMPI, the modulefile would look something like:

#%Module 1.0
#
#  OpenFOAM v1906 compiled with OpenMPI module file
#


setenv  OPENFOAM_HOME /opt/drz/apps/openfoam/v1906-openmpi/
setenv  OPENFOAM_VERSION v1906

module load openmpi/4.1.5

setenv DRZ_OPENFOAM_MPI_TYPE openmpi
setenv DRZ_OPENFOAM_OMPI_MPIRUN_COMMAND mpirun
setenv DRZ_OPENFOAM_OMPI_MPI_COMMAND_OPTIONS "-mca pml ucx --mca btl ^vader,tcp,openib --mca opal_set_max_sys_limits 1 -x UCX_NET_DEVICES=mlx5_ib0:1"
setenv DRZ_OPENFOAM_OMPI_OMPI_PREFIX  $env(OMPI_PREFIX)

info

The above example is for Microsoft Azure when using the InfiniBand interconnect with OpenMPI. Adjust DRZ_OPENFOAM_OMPI_MPI_COMMAND_OPTIONS as needed to match your hardware.

tip

We recommend getting assistance from HPCBOX support at boxhelp[at]drizti.com for building, fine-tuning and setting up OpenFOAM. We are here to help!