Page History

...

...

This example runs PAUP on the input file input.nex that resides in the current working directory. A file (here we'll name it pbsjob) is created with the contents:

...

Important:  Most software will only consume 1 CPU core - e.g., requesting 8 CPU cores for a PAUP job blocks other people using the unused 7 CPU cores.  Example 1 below would likely be the most users should be basing their job scripts from.  If in doubt, contact HPRC staff.

For more information about PBSPro please click to see guide.  For a brief description of PBS directives provided in examples below, see the "Brief Explanation of PBS directive used in examples above" section immediately following the final example PBS script.

HPC staff should be able to assist researchers needing help with PBS scripts.

Singularity Example

The following PBS script requests 1 CPU core, 2GB of memory, and 24 hours of walltime

#!/bin/bash
#PBS -c s
#PBS -j oe
#PBS -m ae
#PBS -N jobnameJobName1
#PBS -M jc123456@jcuFIRSTNAME.LASTNAME@jcu.edu.au
#PBS -l walltime=200024:00:00

echo "------------------------------------------------------"
echo " This job is allocated 1 cpu on "
cat $PBS_NODEFILE
echo "------------------------------------------------------"
echo "PBS: Submitted to $PBS_QUEUE@$PBS_O_HOST"
echo "PBS: #PBS -l select=1:ncpus=1:mem=2gb

cd $PBS_O_WORKDIR

# Add this to all of your scripts
shopt -s expand_aliases
source /etc/profile.d/modules.sh
 
# Output some useful information about
# the job we are running
echo "Job identifier is $PBS_JOBID"
echo "Working directory is $PBS_O_WORKDIR"
echo "PBS: Job identifier is $PBS_JOBID"
echo "PBS: Job name is $PBS_JOBNAME"
echo "------------------------------------------------------"
 
# Load the container we want to use
# It is a good idea to always specify the version number 
module load R/4.1.2u1

# Run your code
R ...    # Replace ... with your arguments & options.

Module Examples

#!/bin/bash
#PBS -j oe
#PBS -m ae
#PBS -N JobName1
#PBS -M FIRSTNAME.LASTNAME@jcu.edu.au
#PBS -l walltime=24:00:00
#PBS -l select=1:ncpus=1:mem=2gb

cd $PBS_O_WORKDIR
shopt -s expand_aliases
source /etc/profile.d/modules.sh
echo "Job identifier is $PBS_JOBID"
echo "Working directory is $PBS_O_WORKDIR"

module load paup/4b10
paup -n input.nex

To submit the job for execution on a HPRC compute node simply enter the command:

qsub pbsjob

If you know this job will require more than 4GB but less than 8GB of RAM, you could use the command:

qsub -l nodes=1:ppn=2 pbsjob

If you know this job will require more than 8GB but less than 16GB of RAM, you could use the command:

qsub -l nodes=1:ppn=8 pbsjob

The reason for the special cases (latter two) is to guarantee memory resources for your job. If memory on a node is overallocated, swap will be used. Job(s) that are actively using swap (disk) to simulate memory could take more than 1000 times longer to finish than a job running on dedicated memory. In most cases, this will mean your job will never finish.

...

Using Job Arrays

Users with a knowledge of shell scripting (e.g., bash) may choose to take advantage of job arrays. This feature significantly reduces load on our Torque/Maui server (compared to lots of individual job submissions). The example below (assume the file name is pbsjob), will only be useful as a guide

#!/bin/bash
#PBS -j oe
#PBS -m ae
#PBS -N JobName3
#PBS -M FIRSTNAME.LASTNAME@my.jcu.edu.au
#PBS -l walltime=240:00:00
#PBS -l select=1:ncpus=20:mem=60gb

cd $PBS_O_WORKDIR
shopt -s expand_aliases
source /etc/profile.d/modules.sh
echo "Job identifier is $PBS_JOBID"
echo "Working directory is $PBS_O_WORKDIR"

module load migrate
module load mpi/openmpi
mpirun -np 20 -machinefile $PBS_NODEFILE migrate-n-mpi ...

#!/bin/bash
#PBS -j oe
#PBS -m ae
#PBS -N JobName2
#PBS -M FIRSTNAME.LASTNAME@my.jcu.edu.au
#PBS -l walltime=3:00:00
#PBS -l select=1:ncpus=8:mem=32gb

cd $PBS_O_WORKDIR
shopt -s expand_aliases
source /etc/profile.d/modules.sh
echo "Job identifier is $PBS_JOBID"
echo "Working directory is $PBS_O_WORKDIR"

module load matlab
matlab -r myjob1 &
matlab -r myjob2 &
matlab -r myjob3 &
matlab -r myjob4 &
matlab -r myjob5 &
matlab -r myjob6 &
matlab -r myjob7 &
matlab -r myjob8 &
wait    # Wait for background jobs to finish.

#!/bin/bash
#PBS -

j oe
#PBS -m ae
#PBS -N 

ArrayJob
#PBS -M 

FIRSTNAME.LASTNAME@jcu.edu.au
#PBS -l walltime=

1:20:00

#PBS -l select=1:ncpus=1:mem=1gb

cd $PBS_O_WORKDIR
shopt -s expand_aliases
source /etc/profile.d/modules.sh

module load matlab
matlab -r myjob$PBS_ARRAYID

Issuing the command qsub -S /bin/bash -t 1-8 pbsjob will see 8 jobs run under one major identifier. To view status of individual jobs in the array. The above example is identical to the one in the "Do It Yourself" section below.

Chances are you may need more advanced features of the scripting language than what is shown above. HPRC staff will endeavour to provide assistance with job arrays, if requested.

Do It Yourself

There are several legitimate reasons for wanting to run multiple single processor jobs in parallel within a single PBS script. For example, you may want to run 8 MATLAB jobs which require a toolbox that only has 4 licensed users. Only 1 MATLAB license is checked out if all 8 jobs are run on the same system. An example PBS script to do this task would look like

Example 5:

The following script is a rework of Example 2 to use the /fast/tmp filesystem for a hyperthetical workflow that is I/O intensive.  This example assumes 1 output file per job.

#!/bin/bash
#PBS -c s
#PBS -j oe
#PBS -m ae
#PBS -N jobnameJobName2
#PBS -M jc123456@jcuFIRSTNAME.LASTNAME@my.jcu.edu.au
#PBS -l walltime=9993:00:00
#PBS -l nodesselect=1:ppnncpus=8:mem=32gb

echo "------------------------------------------------------"
echo " This job is allocated 8 cpus on "
cat $PBS_NODEFILE
echo "------------------------------------------------------"
echo "PBS: Submitted to $PBS_QUEUE@$PBS_O_HOST"
echo "PBS: cd $PBS_O_WORKDIR
shopt -s expand_aliases
source /etc/profile.d/modules.sh
echo "Job identifier is $PBS_JOBID"
echo "Working directory is $PBS_O_WORKDIR"

echo "PBS: Job identifier is $PBS_JOBID"
echo "PBS: Job name is $PBS_JOBNAME"
echo "------------------------------------------------------"

cd $PBS_O_WORKDIR
source /etc/profile.d/modules.shmkdir -p /fast/tmp/jc012345/myjobs
cp -a myjob1.m myjob2.m myjob3.m myjob4.m myjob5.m myjob6.m myjob7.m myjob8.m /fast/tmp/jc012345/myjobs/
pushd /fast/tmp/jc012345/myjobs

module load matlab
matlab -r myjob1.m &
matlab -r myjob2.m &
matlab -r myjob3.m &
matlab -r myjob4.m &
matlab -r myjob5.m &
matlab -r myjob6.m &
matlab -r myjob7.m &
matlab -r myjob8.m &
wait    # Wait for background jobs to finish.

cp -a out1.mat out2.mat out3.mat out4.mat out5.mat out6.mat out7.mat out8.mat $PBS_O_WORKDIR/

To submit the job for execution on a HPRC compute node simply enter the command:

qsub pbsjob

Note: The echo commands in the PBS script example above are informational only.

popd
rm -rf /fast/tmp/jc012345/myjobs

Consider the possibility that you may be running more than one workflow at any given time.  Using subdirectories is a good way of segregating workflows (at a storage layer).

Brief Explanation of PBS directive used in examples above

Brief Details on some extra PBS/Torque directives

For full details on directives that can be used, use "man qsub" on a HPC login node or look at online documentation for Torque.

PBS/Torque Variables

The following variables can be useful within your PBS job script.  Some are present in the examples above.