Environment modules are used on HPC to allow multiple versions of any given piece of software to sit on the same OS, giving you the ability to choose a version of software.
HPC cluster nodes have some convenient aliases configured (26-Jun-2017):
...
A Brief Introduction to using environment modules
To view software under environment modules control, use the command:
| No Format |
|---|
module avail |
To list the environment modules you have loaded, use the command:
| No Format |
|---|
module list |
To load the contents of environment module (easier use of software) use the module load command. For example, to load the contents of the module associated with the latest version of Oracle Java/JDK 1.8.0, you can use:
| No Format |
|---|
module load java/1.8.0-oracle |
If you want to change the version of software you wish to use, the module switch command can be used. For example, lets say that you have already loaded python/2.7.13 and want to switch to python/3.6.1. This can be done with:
| No Format |
|---|
module switch python/3.6.1 |
To unload a module, use the module unload command. For example:
| No Format |
|---|
module unload java/1.8.0-oracle |
will backout changes made to your environment from a previous module load java/1.8.0-oracle command. To unload (purge) all modules you have loaded, use the command:
...
All HPC facilities, worldwide, provide software in a very different way to personal and corporate computing platforms. JCU HPC uses environment modules to deliver multiple versions of software to researchers using our multi-user HPC cluster. This approach is widely used for the following reasons:
- Most operating systems weren't written to deal with having multiple versions of the same software available. Even in cases where multiple versions are installed, e.g., java, it is left to a system administrator to set the version which everyone will use.
- Using environment modules, Individual users or groups can determine which version of software they wish to use.
- Loading of modules with version information maximizes your chance of being able reproduce your results upon demand.
- In the corporate world, system administrators have to configure a default version of piece of software (e.g., java) that everyone will use.
- Using environment modules also improves performance by reducing the length of your search path (executable of interest found sooner). It also reduces the chance of you having to deal with application name conflicts.
In 2021, HPC staff commenced a major project to replace all existing software (including conda environments) with containerized versions of the same software and/or environments. The container delivery platform being using is Apptainer (singularity) which mirrors the approach that all other HPC facilities are using or moving toward using. The move to containerized HPC is being driven by a requirement for all research to be done in a reproducible way.
| Expand | ||
|---|---|---|
| ||
For security reasons, all software installed as part of the OS must be kept up to date.
Software installed using the traditional, compile from source, method comes with serious consequences to reproducible research.
Operating system and software providers release software with a defined lifecycle.
In many cases, the above points mostly relate to risk of work not being reproducible. Moving to containerized delivery of software provides a guarantee of reproducibility, assuming that the container platform continues to be available and supported. |
Environment Modules Cheat Sheet.
Information Requests
You will need to replace any occurrence of <software-name> and/or <version> below with an appropriate name/value.
| Code Block | ||
|---|---|---|
| ||
module avail |
| Code Block | ||
|---|---|---|
| ||
module avail <software-name>
# A partial match will be sufficient. |
| Code Block | ||
|---|---|---|
| ||
module help <software-name> |
| Code Block | ||
|---|---|---|
| ||
module show <software-name>/<version>
# For apptainer based software, you will see aliases that have been created for ease of use. |
| Code Block | ||
|---|---|---|
| ||
module list |
Modification of your environment (paths & aliases)
You will need to replace any occurrence of <software-name> and/or <version> below with an appropriate name/value.
| Code Block | ||
|---|---|---|
| ||
module load <software-name>/<version>
# Omission of "/<version>" will result in the default version (changes over time) being loaded.
# Note that the default version may be the "safest", rather than the latest, version. |
| Code Block | ||
|---|---|---|
| ||
module switch <software-name>/<version> |
| Code Block | ||
|---|---|---|
| ||
module unload <software-name>
# The specification of a version shouldn't be required. |
| Code Block | ||
|---|---|---|
| ||
module purge |
Information for researchers using conda environments
Environment module files for conda environments have been removed, since discovery of a few conda environments that couldn't be reinstalled. After you issue the command
| No Format |
|---|
module load conda3
|
you will be able to use the conda command to list environments available.
Notes for future
Conda environments will not be available on future HPC platforms.
There is increasing pressure to move HPC workloads into the public cloud. Reliance on in-house HPC infrastructure will decrease with time.
It's possible that HPC will be directed to implement a NOEXEC option on all user filesystems, which would mean software you have installed yourself will not be executable. I would hope that there is a documented security vetting process prior to such a directive being given to us. We have had a request for implementation of such an option in the past but chose not to take any action (there was no directive).