Child pages
  • HPC Best Practices
Skip to end of metadata
Go to start of metadata

There is a significant risk to relying on your desktop/laptop computer and/or USB disks.  Example situations to think about:  Disk failure, corruption of data, power outages.  HPC staff have been alerted to 5 cases where PhD students nearing the end of their candidature have "lost everything" because a disk in their laptop failed.  HPC staff have also been asked to try and recover files from a faulty USB disk - not always possible.  JCU perform regular RCD testing - expect to lose power from time to time.

HPC Cluster (Linux)

The longer your computational research jobs take, the more you should be aware of the following.

  1. Submitting jobs (using qsub) to compute nodes is recommended.
  2. If you must work interactively, learn how to use the screen command in Linux.  Jobs running interactively will be killed by scheduled security patching that requires a reboot.
  3. Be aware of how much resource your jobs require - particularly the amount of memory (RAM) required.  Overconsumption of memory on a server can result in failiure - all running jobs on that server will need to be re-run (not just yours).

Research requiring Microsoft Windows

If your research computations can only be completed on Micosoft Windows, the following points are worth noting.

  1. Identify the minimum resource requirement (CPU, memory, and disk space) for your work.
  2. Request a resource (virtual machine) - HPC will be able to cater for many requirements.  Use Remote Desktop Connection to work on that resource.

Research Data Protection

If your research produces data/information that can never be recaptured (e.g., photos/videos of "events"), make every effort to protect it - you cannot travel back in time.

  1. Keep one copy of all research data on HPC.  Feel free to have one or more copies elsewhere (e.g., on your desktop/laptop computer).
  2. Establishing a regular (e.g., weekly) synchronisation of data is best.  This process should only update changes.
  3. Avoid deleting your files, or think long and hard before doing so.  HPC storage is not backed up.

All data placed on HPC is replicated to an off-site location, but this does not protect against accidental deletion.

Backups

Unfortunately, HPC does not have sufficient funding to provide a regular backup target for research data. 

For small amounts of data, you could backup your data to OneDrive, DropBox, or similar.  For large amounts of data, be sure to properly evaluate ongoing budget requirements before submitting to a solution.  E.g., The cost of recalling data from long-term archive/backup locations may be much higher than the costs of storing the data.

  • No labels