Compute Canada

How to setup/submit a Fluent parallel production job?

Please note: The FAQ pages at the HPCVL website are continuously being revised. Some pages might pertain to an older configuration of the system. Please let us know if you encounter problems or inaccuracies, and we will correct the entries.

To submit a production job on HPCVL clusters, you need to us the load-balancing software Grid Engine. To obtain details, read our gridengine FAQ.

For a Fluent production job, this means that rather than issuing the above batch command directly, you wrap it into a Grid Engine script that looks somewhat like this:

#!/bin/bash
#$ -S /bin/bash
#$ -V
#$ -cwd
#$ -pe shm.pe {number of processors}
#$ -m be
#$ -M {email address for notification}
#$ -o {full name of output file}
#$ -e {full name of error file}
{additional commands, eg for removing files}
. /opt/fluent/ansys-14.0/setup_64bit.sh
fluent {2d/3d/2ddp/3ddp} -t$NSLOTS -g -i {input file}

This script needs to be altered by replacing all the relevant items enclosed in {} by the right values. A Unix text formatet version of the above template can be found here.

The advantage to submit jobs via a load balancing software is that the software will automatically find the resources required and put the job onto a set of processors that have a low load. This will help executing the job faster. Note that the usage of Gridengine for all production jobs on HPCVL clusters is mandatory. Production jobs that are submitted outside of the load balancing software will be terminated by the system administrator.

The Fluent jobs that you will want to run on the HPCVL machines are likely to be quite large. To utilize the parallel structure of our machines, Fluent offers several options to execute the solver in a parallel environment, i.e. on several CPU's simultaneously. The default option for such runs is MPI i.e., it uses the Message Passing Interface for inter-process communication.

To take advantage of the parallel capabilities of Fluent, you have to call the program with additional commandline options that specify the details of your parallel run:

  • -tn where n is the number of processors requested, e.g. if you want to run with 8 processors, you would use the option -t8
  • -g specifies that the GUI should be surpressed. Required for batch jobs.

Parallel jobs should only be run in batch using the Grid Engine. The number of processors specified in our example script appears only once, after

#$ -pe shm.pe

which is where you let the Gridengine know how many processors to allocate to run the program. The internal environment variable $NSLOTS will automatically be set to this value and can then be used in the fluent command line.

It is also necessary to source a setup file called setup_64bit.sh. This will set various environment variables and enable the Fluent program to properly interact with Grid Engine. If you are interested, take a look. The file is readable. 

All processes are allocated within a single node. This is to make communication more efficient and to avoid problems with the control by Gridengine. The effect of this is that, while still using MPI, Fluent employs a so-called shared-memory layer for communication. The disadvantage is that the size of the job is restricted by the number of cores on a node.

Once the script has been adapted (let's call it "fluent.sh"), it can be submitted to the Gridengine by

qsub fluent.sh

from the Solaris login node sflogin0 (which is the GridEngine submit host). Note that the job will appear as a parallel job on the GridEngine's qstat or qmon commands. Note also that submission of a parallel job in this way is only profitable for large systems that use many CPU cycles, since the overhead for assigning processes, preparing nodes, and communication between them is considerable.

There is an easier way to do this: We are supplying a small perl script called that can be called directly, and will ask a few basic questions, such as the name for the job to be submitted and the number of processes to be used in the job. Simply type

AnsysSubmit

and answer the questions. The script expects a Fluent input file with "file extension" .flin to be present and will do everything else automatically. This is meant for simple Fluent job submissions. More complex job submissions are better done manually.