Page History

Environment modules are used on HPC to allow multiple versions of any given piece of software to sit on the same OS, giving you the ability to choose a version of software.  When viewed from an individual user point of view this may seem complex, however it is necessary in a multi-user environment.

Note:  It is possible to have two or more versions of the same module loaded, but this cause problems and should be avoided.  The reason for this is often that a major version change in software can result in significantly different syntax, so software developers maintain multiple, independent streams of their software.  All aliases defined above can be considered examples of this.

Listing available software and versions

module avail

Information about a piece of software (example)

module help grass

Listing environment modules you have loaded.

module list

Loading a specific software version (example)

module load gcc/6

Loading the default version of a piece of software (example)

module load java

 NOTE:  The default software version will generally be the safest (e.g., version distributed by RedHat).

An example of a command to switch to another version of the same software is:

module switch java/1.6.0-openjdk

Unloading a module (example)

module unload java

In some cases, you may want/need to add the version.

Unloading all modules

There will be occasions where is easier to make a fresh start, in which case you simply run the command

module purge

Information specific to JCU HPC configuration

Most module files, when loaded, will set an environment variable <SOFTWARE>_HOME which holds the installation home for the software.  For example, try

module load vcftools
echo $VCFTOOLS_HOME
ls -lR $VCFTOOLS_HOME

The most common environment variable modified by the loading of an environment variable is PATH.

Other commonly modified environment variables are: MANPATH, LD_LIBRARY_PATH, CLFAGS, CXXFLAGS, and LDFLAGS.

Environment modules are not usually created for Python extensions.  Loading a specific python version will usually enable use of all extensions.  Note that some extensions may not be available for all versions of python.

Software that includes perl extensions will usually modify the PERLLIB and/or PERL5LIB environment variables.

For Java based software, there will often be an alias set the simplifies things for users.  For example a

module load picard

will create a picard alias that expands to java -jar $PICARD_HOME/picard.jarAll HPC facilities, worldwide, provide software in a very different way to personal and corporate computing platforms.  JCU HPC uses environment modules to deliver multiple versions of software to researchers using our multi-user HPC cluster.  This approach is widely used for the following reasons:

• Most operating systems weren't written to deal with having multiple versions of the same software available.  Even in cases where multiple versions are installed, e.g., java, it is left to a system administrator to set the version which everyone will use.
• Using environment modules, Individual users or groups can determine which version of software they wish to use.
• Loading of modules with version information maximizes your chance of being able reproduce your results upon demand.
• In the corporate world, system administrators have to configure a default version of piece of software (e.g., java) that everyone will use.
• Using environment modules also improves performance by reducing the length of your search path (executable of interest found sooner).  It  also reduces the chance of you having to deal with application name conflicts.

In 2021, HPC staff commenced a major project to replace all existing software (including conda environments) with containerized versions of the same software and/or environments.  The container delivery platform being using is Apptainer (singularity) which mirrors the approach that all other HPC facilities are using or moving toward using.  The move to containerized HPC is being driven by a requirement for all research to be done in a reproducible way.

Environment Modules Cheat Sheet.

Information Requests

You will need to replace any occurrence of <software-name> and/or <version> below with an appropriate name/value.

module avail

module avail <software-name>
# A partial match will be sufficient.

module help <software-name>

module show <software-name>/<version>
# For apptainer based software, you will see aliases that have been created for ease of use.

module list

Modification of your environment (paths & aliases)

You will need to replace any occurrence of <software-name> and/or <version> below with an appropriate name/value.

module load <software-name>/<version>
# Omission of "/<version>" will result in the default version (changes over time) being loaded.
# Note that the default version may be the "safest", rather than the latest, version.

module switch <software-name>/<version>

module unload <software-name>
# The specification of a version shouldn't be required.

module purge

Information for researchers using conda environments

Environment module files for conda environments have been removed, since discovery of a few conda environments that couldn't be reinstalled.   After you issue the command

module load conda3

you will be able to use the conda command to list environments available.

Notes for future

Conda environments will not be available on future HPC platforms.

There is increasing pressure to move HPC workloads into the public cloud.  Reliance on in-house HPC infrastructure will decrease with time.

It's possible that HPC will be directed to implement a NOEXEC option on all user filesystems, which would mean software you have installed yourself will not be executable.  I would hope that there is a documented security vetting process prior to such a directive being given to us.  We have had a request for implementation of such an option in the past but chose not to take any action (there was no directive).