This is a short introduction into how to carry over code from a serial programming environment to the SUNFire multi-processor system used by HPCVL. It is meant to give the user a basic idea of what to do to get the code running on several processors. We assume that the code is written in FORTRAN, but most considerations carry over directly to C/C++ code. The document is organized in an "FAQ" manner, i.e. a list of "obvious" questions is presented as a guideline. Please feel free to contact Hartmut Schmider if you want to see more questions included.

Frequently Asked Questions:

1.      Where are the Fortran and C/C++ compilers located?

2.      Which environment variables do I have to set, what does my path have to look like if I want to do program development?

3.      How do I compile and link serial programs? Which compiler flags should I use?

4.      Will my serial code run in parallel without changes?

5.      How do I "parallelize" my code?

6.      How can I use multiple threads to get parallel performance out of my serial code?

7.      How do I force multi-thread parallelization? How to use compiler directives?

8.      What is MPI and when do I use it?

9.      How do I parallelize my code with MPI?

10.  How do I compile and run MPI code on the SUN?

11.  How can I check out performance of my serial, multi-threaded, or MPI code?

12.  It doesn't work. Where can I get help?

1.Where are the Fortran and C/C++ compilers located?

On the SUNFires of HPCVL, the Fortran and C++ compilers and the needed headers, libraries and tools can be found under the /opt/studioXX/SUNWspro subdirectory system. XX stands for the version. The current version is 12. The compilers for F77, F90, F95, C and C++, together with a development tool called "sunstudio" are under /opt/studioXX/SUNWspro/bin. Various libraries are under /opt/studioXX/SUNWspro/lib. This includes dynamic ones, so if your program complains about not finding "mickey_mouse.so", setting

  LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/studioXX/SUNWspro/lib 

Back to top...

2.Which environment variables do I have to set, what does my path have to look like if I want to do program development?

On our new cluster, you do not have to add anything to your default setup to use program development tools such as compilers and debuggers. We are using a program called usepackage which replaces the issuing of lengthy setting for environment variables by a simple command "use". Without issuing any additional use commands, you start with standard-user-settings that include the latest compilers and development tools. If you want to change this, you can do so by issuing the use package command, where package stands for one of the following:

   ct6 - Sun ClusterTools 6
   studio12 - Sun Studio 12 Compilers and Tools
   studio11 - Sun Studio 11 Compilers and Tools
   studio10 - Sun Studio 10 Compilers and Tools
   studio8 - Sun Studio 8 Compilers and Tools
   studio7 - Sun Studio 7 Compilers and Tools
   workshop6 - Sun Workshop 6 Compilers and Tools

You can do things manually, of course. You should have something like

   PATH=$PATH:/opt/studio10/SUNWspro/bin:/opt/SUNWhpc/bin 
   export PATH
   MANPATH=$MANPATH:/opt/ss10/SUNWspro/man:/opt/SUNWhpc/man
   export MANPATH

Back to top...

3.How do I compile and link serial programs? Which compiler flags should I use?

You will use the "Sun Studio" compilers which reside in /opt/studioXX/SUNWspro/bin to compile and link. To compile a Fortran 77, Fortran 90, Fortran 95, C, or C++ program, you issue the f77, f90, f95, cc, or CC commands, respectively. Compiling and linking is best done with a makefile. But you can also issue the commands by hand.

To compile:

  compiler -c [options] name.ext

To link:

  compiler -o name [options] [libraries] list 

Using the compilers and the linker in the above manner requires the proper setting of the PATH environment variable.

There are hundreds of compiler flags, and many of them are not required most of the time. A few that are in more frequent use are:

-xOn optimizes your code. nis a number from 1 to 5 with increasing severity of alterations made to the code, but also increasing gain. Up to -xO3 is generally rather safe to use. But you should, of course, always check results against an un-optimized version: they might differ.

-fast is a combination of optimization flags that is quite safe to use and often improves performance a lot. However, the resulting code is specific for the current UltraSparc-IV+ machines and cannot be executed on older SUN's (including the UltraSparc-III based 15K's that are still part of our cluster). Note that this overrides the -xOn option if it comes after it, since compiler options are executed from left to right!

-g produces code that can be debugged. Unlike for other compilers, -g and -xOn are not mutually exclusive, so it is a good flag to have in the development stage of a program.

-v produces more output than you can handle, which makes it easier to track down problems.

-lname is used to bind in a library called libname.a (static) or libname.so (dynamic). This flag is used to link only.

-Ldirname is used in conjunction with -lname and lets the linker know where to look for libraries. dirnameis a directory name such as /opt/studio12/SUNWspro/prod/lib.

-Rdirname is used to tell the program where to get dynamic libraries at runtime.

There are many more flags. They are documented in the man pages (man f90 or man cc), as well in various documents that may be downloaded in pdf format from the Sun documentation website. The latter is a good place to look to resolve problems in any case. Use the search engine to obtain User's Guides and Reference Manuals.

Some compiler flags are only useful for parallel programs and will be discussed later. Sometimes there is a considerable performance gain from using specific options (such as -xchip and -xtarget), but the code becomes less general.

Back to top...

4.Will my serial code run in parallel without changes?

No. At the very least, you will have to recompile it with "parallel options" and to set a few environment variables. For most code, that will not be enough either. Fortunately, in many cases, it is not difficult to get the compiler to produce code that will show some performance gain from multi-threading.

Back to top...

5. How do I "parallelize" my code?

In essence there are 4 steps that should be considered to "parallelize" your code:

1. Optimize the serial version as much as you can. Try to make it as "simple" as possible, avoiding nested loops and loops with dependencies, i.e. where the operations inside one iteration depend on the results from a previous one. Dependencies may be hidden in function calls or by reference to global variables or COMMON's. Often, a program spends most of the execution time in a few loops. Those are candidates for parallel performance. Try to find them (e.g. by running analyzer software, such as is available inside the "sunstudio" development tools, or by explicitely inserting timing routines like etime() into the code). Focus on the simplification of those loops.
2. Use auto-parallelization flags of the compiler (see section 6)
3. Force multi-threading via OpenMP compiler directives (see section 7)
4. Use MPI routines to run separate processes that communicate with each other (see section 8).

Back to top...

6.How can I use multiple threads to get parallel performance out of my serial code?

The compilers running on the Sunfire cluster have options that cause it to attempt to parallelize loops that have no dependencies by "multi-threading" them. The compiler flags to get this done are

• -autopar identifies loops that are obviously non-dependent and creates multithreaded code for them
• -reduction reduces the elements of arrays into single values, for example by summing over them
• -loopinfo shows which loops were parallelized, and which not (and why)
• -stackvar Necessary. Allocates local variables on the stack.

Back to top...

7. How do I force multi-thread parallelization? How to use compiler directives?

The compiler will be very conservative about multithreading loops. If there is the slightest possibility of data dependencies, it will refuse to do it if -autopar is used. Function calls within loops, if statements that depend on variables which change in the loop, and many other features will be considered "dangerous" and inhibit parallelization. The reason is that such features have a potential to make the result dependent on the order in which the loop iterations are carried out, and therefore go against a parallel execution.

However, often you know more than the compiler. You might be certain that a function call does not alter the value of variables that are shared with other loop iterations. If this is the case, there is ways to tell the compiler to parallelize anyhow. This is done via compiler directives that look like comments, but if compiled with the proper flags, will guide the compiler in parallelizing the code. The most common one a OpenMP compiler directives. Here is an example:

   !$OMP DO PRIVATE(a)
   do i = 1, n
     a(1) = b(i)
     do j = 2, n
       a(j) = a(j-1) + b(j) * c(j) 
     end do
     x(i) = f(a) 
   end do

Some of the compiler flags for this approach are:

• -xexplicitpar parallelize when I tell you to by compiler directives
• -vpara verbose output about dependencies in the explicitely parallelized loops
• -parallel same as -xexplicitpar, but additional autoparallelize if possible
• -mp=type specify type of directives, type can be "sun", "cray" or "openmp"; note that omp directives use OpenMP platform-independent compiler directives which are the de-facto industry standard. The use of these is strongly encouraged.
• -xopenmp includes all necessary flags for usage of OpenMP compiler directives, and therefore replaces the -xexplicitpar -mp=omp combination. Also includes some other flags (see an pages).

Note that a separate OpenMP FAQ is available that contains more information about this programming technique.

Back to top...

8.What is MPI and when do I use it?

The ultimate parallelization is, of course, achieved by re-writing the code in a parallel fashion, so that it can be executed on several separate processors, or indeed machines, separately. For this, it is necessary to establish some communication between the processes, and this is usually done by some form of message passing. A platform independent standard for this is a set of more than 200 routines, available in Fortran and C, that comprise the MPI (Message Passing Interface) standard. Using these routines requires a little rethinking of the code structure, but is in many cases rather simple and effective.

MPI is best used if your code has a good potential to employ many processors independently with none sitting idle. It is also advantageous to have only relatively little communication being necessary between processes. Examples are numerical integration (where independent evaluations of the integrant can be done separately), Monte-Carlo methods, finite-difference and finite-element methods (if the problem can be divided up into blocks of equal size with minimal communication). MPI requires some serious re-coding in some cases, but with a relatively small number of routines, great scaling can be achieved.

Back to top...

9.How do I parallelize my code with MPI

A very simple example of how to parallelize code with MPI is given in the monte.f Fortran program.

Only a few MPI commands are necessary to parallelize this Monte-Carlo calculation of pi. The first

  call MPI_INIT(ierr) 

  call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
  call MPI_COMM_SIZE(MPI_COMM_WORLD, np, ierr)

  call MPI_REDUCE(pi,pisum,1,MPI_DOUBLE_PRECISION,&
       MPI_SUM,0,MPI_COMM_WORLD,ierr)

  call MPI_FINALIZE(ierr)

To get an idea of how to use MPIand what the various routines do, check out the MPI workshop at the Maui HPC Centre site. For a list of routines in the MPIstandard, and a reference manual of their usage, go to the Sun Documentation Website and search for the Sun MPI Programming and Reference Guide . Note that we offer a separate MPI FAQ with more information about this system.

Although the MPI standard comprises hundreds of routines, you can write very stable and scalable code with only a dozen or so routines. In fact, often the simpler you keep it the better it will work.

Back to top...

10.How do I compile and run MPI code on the SUN?

To use MPI on our Sunfire cluster, you will have to do the following things:

• Include header files on the top of all subroutines that use MPI, i.e.
  for Fortran INCLUDE 'mpif.h' and for C/C++ #include <mpi.h>
  This is important for the definition of variables and constants that are used by the MPI system.
• Compile and link with the following flags:
  -I/opt/SUNWhpc/include -L/opt/SUNWhpc/lib -R/opt/SUNWhpc/lib -lmpi
  These tell the compiler, linker and runtime environment where to look for include files, static libraries and runtime dynamic libraries. The command -lmpi loads the MPI library.
• Alternatively to the above flags, you can use the tmf90, tmcc, or tmCC macros for Fortran, C, and C++, respectively, instead of the standard compilers/linkers. These will automatically call the right flags. However, the -lmpi library flag still has to be issued.
• For running MPI programs, a special multi-processor runtime environment (CRE) is needed. This allows you to specify how many processes are used for the execution of the program, from which pool of processes they should be taken, etc...

file:///opt/studioXX/SUNWspro/prod/lib/locale/C/html/index.html