Child pages
  • HPC Performance Considerations

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following points/suggestions should be considered when running jobs concurrently:

  • HPRC HPC storage was has been purchased in attempt to meet capacity requirements (it was a critical issue at the time).
  • If you are reading/writing large files (greater than 10MB say), limit the number of concurrent jobs you are running (less than 10 at a time).  Users running concurrent tar, zip, gzip, and/or bzip2 commands are the reason for most incidents where people complain about 'slow storage'.
  • Avoid using parallel utilities like pbzip2, when running concurrent jobs.
  • Create a separate job (sequential) for all your post-run I/O transactions (such as those mentioned above)deliver expected capacity requirements - evaluation of vendor responses is dominated by $/TB. 
  • Our current (2021) general use storage platform is very much entry level capabilities when it comes to performance.
  • Researchers should avoid running parallel/concurrent tar, gzip, etc. commands.  Utilities such as pbzip2 should also be avoided.
    • If your computational research is I/O intensive, ensure that it is configured to use local scratch space (/fast)
    • Consider other researchers and create a separate, sequential operations job for all your post-job I/O transactions.
  • Compression tools don't provide much benefit when working on binary data files - it's generally better to leave binary files uncompressed.

...