Compute Canada

PVM

Please note: The FAQ pages at the HPCVL website are continuously being revised. Some pages might pertain to an older configuration of the system. Please let us know if you encounter problems or inaccuracies, and we will correct the entries.

What is PVM?

PVM (Parallel Virtual Machine) is a message passing programming system, designed to link separate host machines to form a ``virtual machine'' which is a single, manageable computing resource. The PVM system is portable to a wide variety of architectures including workstations, multiprocessors and supercomputers.

PVM is a Public domain software package developed at Oak Ridge National Laboratory, USA. It is Comprised of two main components - the PVM daemon process (pvmd3) and the library interface routines (libpvm3.a, libfpvm3.a & libgpvm3.a). The distribution installed is the SunFire is pvm version 3.4.3 with architecture set to SUN4SOL2 which means that PVM will performs its communications via sockets.

Why PVM and not MPI?

There is a large literature about the question of PVM vs MPI. We present here some of PVM features that may let you consider using PVM. PVM supports a higher level of portability that is extended to interoperability between different languages, messages can be sent and received between C and Fortran programs for example. Another feature of PVM is that it allows the development of fault tolerant applications that can survive host or task failure and provide notification to the user in this event, which could be a critical issue for large applications or long-time running simulations. PVM provides also a dynamic resource management and process control functions, user can add or delete hosts at will, giving a flexible control over the amount of computational power being utilized, thus leading to more efficiency.

What languages does PVM work with?

PVM is independent of any particular language (it is written in C). Programs can be written in one or more languages. Libraries for C, C++ and Fortran are provided with the distribution. A PVM code can be ported heterogeneously to run cooperatively across any set of different architectures with different languages at the same time.

How do I start using PVM?

Before starting with PVM you must define the variable PVM_ROOT which indicates where the PVM system is installed, and also the variable PVM_ARCH which indicates the platform where PVM is running to distinguish between machines that runs different excutables. For this purpose, one must set the variables PVM_ROOT to /usr/local/pvm3 andPVM_ARCH to $PVM_ROOT/lib/pvmgetarch in his startup files.

A better way to define this variables is by concatenating to the file .cshrc, .bashrc, .kshrc the content of the file $PVM_ROOT/lib/cshrc, bashrc, kshrc.stub. The stub should be placed after PATH and PVM_ROOT are defined, it will automatically determine the type of architecture it is on and set up environment variables for PVM to use. It is particularly useful when the user runs his PVM applications on different architectures.

You should also set the MANPATH to be able to read the on-line manual pages

setenv MANPATH $PVM_ROOT/man:$MANPATH (tcsh or csh) export MANPATH=$PVM_ROOT/man:$MANPATH (bash or ksh)

How do I compile my PVM code?

Your PVM applications must contain the header files that defines all the functions and constants used by the PVM. Programs written in C should include header file pvm3.h, as follows

#include

The Fortran programs should include header file fpvm3.h as follows

#INCLUDE 'fpvm3.h'

- In order to create your PVM executables, you must compile and link them with th appropriate libraries. For example the command

cc/f77/CC [compiler flags] [source files] [loader flags] -L$PVM_ROOT/lib/$PVM_ARCH -I$PVM_ROOT/include -lfpvm3 -lgpvm3 -lpvm3 [libraries needed by PVM] [other libraries]

will compile your program and link it to all needed libraries and include files.

The -lpmv3 is always required in the compile command,

the -lfpvm3 flag is required only in case you are using Fortran,

the -lgpvm3 flag is required only in case you are using dynamic groups.

- A more portable and efficient way to compile your programs is the use of the aimk makefile utility that come with PVM, it is a wrapper program for the UNIX make that determines the PVM machine architecture and sets the ARCHLIB to the necessary system libraries. To keep a portable environment for your PVM applications you can create your own portable aimk makefile called Makefile.aimk and use it to build and compile your codes.

How do I run my PVM code?

To get PVM running, you must start either a pvm deamon pvmd or the PVM console by hand. For the moment let's focuse on how to run PVM from the command line, the next sections will deal with how to run PVM applications as batch jobs via Grid Engine. To start PVM console do the following

%pvm pvm_hostfile

Where pvm_hostfile is a file used to customize your PVM system for a particular application or environment. The hostfile contains at least the list of the hosts (nodes) you intend to use. For example in the case of SunFire nodes you can add this to your hostfile

sfnode0

sfnode1

sfnode2

sfnode3

You can also indicate in the hostfile your working directory, the locations of your executables, etc... . An example of a pvm hostfile can be found in /usr/local/pvm3/pvm_hostfile

After PVM is running, you can spawn your program to the desired number of processes from the pvm console

pvm> spawn -nslots -> executable

or you run your executable from the Unix command line, like any other program.

How do I setup my environment to run PVM with GridEngine?

Once your PVM application is debugged and ready for running, we recomand that you send your job in batch mode via Grid Engine. For this purpose you have to add some environment variables needed for the integration between PVM and Grid Engine.

First, make sure that the Grid Engine settings are already defined in your .cshrc or .bashrc file. Otherwise add one of the following lines to your environment (see Grid Engine FAQ)

source /opt/codine52/default/common/settings.csh (for csh or tcsh)

. /opt/codine52/default/common/settings.sh (for ksh or bash)

After that set the folloing environment variables so that GE could export them to all the nodes running your PVM application.

export PVM_ROOT=/usr/local/pvm3

export PVM_RSH=$CODINE_ROOT/pvm/pvm_rsh

export PVM_PATH=:

export PVM_WD=

If this variables are included in your .cshrc or .bashrc don't forget to export them with Grid Engine via the command ``qsub -V'' while submitting your job, or with setting ``#$ -V'' in your batch script file.

You can also define this environment variables in the pvm batch script file and export them via the setting ``#$ -V'' as shown in the next section.

How do I submit my PVM jobs with Grid Engine?

GridEngine provides a flexible interface that handles parallel jobs running on the top of PVM system. A parallel environment called pvm is already defined, as well as pvm queues. The pvm PE contains all information needed by GE to start the parallel job, launch the pvm deamons on the nodes given by a predefined hostfile (ususally it will consiste on sfnode0, sfnode1, sfnode2 and sfnode3), and shut down the PE processes including the pvm deamons once your program finishes or is aborted.

- Setting your pvm bathc script file

Your pvm job script will start like all GE job scrips with the line

#!/bin/sh

That indicate the shell your are using. After, and if it is not already done in your startup file, define the PVM environment variables PVM_ROOT and PVM_RSH.

export PVM_ROOT=/usr/local/pvm3

export PVM_RSH=$CODINE_ROOT/pvm/pvm_rsh

You need to add your hostfile information about your working directory, the location of your executables and and of pvm executables too.

export PVM_PATH=$HOME/pvm_path/$PVM_ARCH:/usr/local/pvm3/bin/$PVM_ARCH

export PVM_WD=$HOME/pvm_path:

and then add on of the settings

#$ -V

#$ -v PVM_ROOT,PVM_RSH,PVM_WD,PVM_PATH

to informe Grid Engine to export all your environment variables to the context of the parallel job.

- The parellel environment pvm defines also specific pvm queues, and in order to force PVM jobs to PVM queues you must add the setting #$ -l pvm in your script file or submit your job with ``-l pvm" option.

- Ask Grid Engine to start the pvm parallel environment (PE) with a specified number of slots.

#$- pe pvm

Where NSLOTS is the number of slots requested for this job. You can also specifie different number of available slots with

#$ -pe pvm 32,16,8,4-1

in that case Grid Engine will allocate the appropriate resources as available, it will provide at most 32 slots and at least one slot for the job.

Finally, if your PVM executables are in different directories, we recomende that you use the absolute path for them in the job script file

$HOME/$PVM_PATH/ $NSLOTS

Once your PVM job script file is ready launch your job with the command

% qsub -V -pe pvm -l pvm pvm_batchfile

A detailed pvm batch file can be found in $CODINE_ROOT/pvm/examples/batch.pvm

You can also submit your pvm job via the qmon GUI. Click on the Job Submission submenu and then enter the name of the batch file in the job script icon.

Is there any published or online material related to PVM?

Yes. The book, PVM: Parallel Virtual Machine - A Users' Guide and Tutorial for Networked Parallel Computing, by Al Geist et. al. is available in good booksores and is also available in postscript and html from netlib at http://www.netlib.org/pvm3/index.html. You will also find a more comprehensive FAQ and a quick reference card about PVM in the same site.

There is also a directory on netlib for contributed software related to PVM.

Finally one should consulte the on-line manual pages included with the distribution, and eventually read and post to newsgroup comp.parallel.pvm.