Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

In 2021, HPC staff commenced a major project to replace all existing software (including conda environments) with containerized versions of the same software and/or environments.  The container delivery platform being using is Apptainer (singularity) which mirrors the approach that all other HPC facilities are using or moving toward using.  The move to containerized HPC is being driven by a requirement for all research to be done in a reproducible way.

Expand
titleNon-reproducibility reproducibile? (examples)

For security reasons, all software installed as part of the OS must be kept up to date.

  • This has potential to impact your computational research results.
  • Most researchers would be unaware of the behind the scenes changes or what impacts they might have had.
  • It would be nigh on impossible for me to recreate the state of HPC cluster software operating systems at any nominated point in time.

Software installed using the traditional, compile from source, method comes with serious consequences to reproducible research.

  • Such installs usually result in a software environment with a potentially very large number of unknown dependencies.
  • Once again, this makes it nigh on impossible to reproduce at a later date.

Operating system vendor release operating systems and software will and software providers release software with a defined lifecycle.

...

  • When RedHat Enterprise Linux 6 (RHEL6) went end of life, we moved to RHEL7.  All software had to be recompiled/reinstalled.
  • Obtaining a copy of RHEL6 would be very difficult.  Being given security clearance to use it would be even more difficult.
  • After python 2 hit end of life a few environments that required python 2, indirectly, could not be reinstalled (dependency not available).

In many cases, the above points mostly relate to risk of work not being reproducible.  Moving to containerized delivery of software provides a guarantee of reproducibility, assuming that the container platform continues to be available and supported.

Environment Modules Cheat Sheet.

You will need to replace any occurrence of <software-name> and/or <version> below with an appropriate name/value.


Code Block
titleListing all available software
module avail

To list all available software

...