Compute Canada

How do I submit my PVM jobs with Grid Engine?

Please note: The FAQ pages at the HPCVL website are continuously being revised. Some pages might pertain to an older configuration of the system. Please let us know if you encounter problems or inaccuracies, and we will correct the entries.

GridEngine provides a flexible interface that handles parallel jobs running on the top of PVM system. A parallel environment called pvm is already defined, as well as pvm queues. The pvm PE contains all information needed by GE to start the parallel job, launch the pvm deamons on the nodes given by a predefined hostfile (ususally it will consiste on sfnode0, sfnode1, sfnode2 and sfnode3), and shut down the PE processes including the pvm deamons once your program finishes or is aborted.

- Setting your pvm bathc script file

Your pvm job script will start like all GE job scrips with the line

#!/bin/sh

That indicate the shell your are using. After, and if it is not already done in your startup file, define the PVM environment variables PVM_ROOT and PVM_RSH.

export PVM_ROOT=/usr/local/pvm3

export PVM_RSH=$CODINE_ROOT/pvm/pvm_rsh

You need to add your hostfile information about your working directory, the location of your executables and and of pvm executables too.

export PVM_PATH=$HOME/pvm_path/$PVM_ARCH:/usr/local/pvm3/bin/$PVM_ARCH

export PVM_WD=$HOME/pvm_path:

and then add on of the settings

#$ -V

#$ -v PVM_ROOT,PVM_RSH,PVM_WD,PVM_PATH

to informe Grid Engine to export all your environment variables to the context of the parallel job.

- The parellel environment pvm defines also specific pvm queues, and in order to force PVM jobs to PVM queues you must add the setting #$ -l pvm in your script file or submit your job with ``-l pvm" option.

- Ask Grid Engine to start the pvm parallel environment (PE) with a specified number of slots.

#$- pe pvm

Where NSLOTS is the number of slots requested for this job. You can also specifie different number of available slots with

#$ -pe pvm 32,16,8,4-1

in that case Grid Engine will allocate the appropriate resources as available, it will provide at most 32 slots and at least one slot for the job.

Finally, if your PVM executables are in different directories, we recomende that you use the absolute path for them in the job script file

$HOME/$PVM_PATH/ $NSLOTS

Once your PVM job script file is ready launch your job with the command

% qsub -V -pe pvm -l pvm pvm_batchfile

A detailed pvm batch file can be found in $CODINE_ROOT/pvm/examples/batch.pvm

You can also submit your pvm job via the qmon GUI. Click on the Job Submission submenu and then enter the name of the batch file in the job script icon.