The following table shows bulk storage appliances purchased for JCU research.

Owner,OperatorUsable SizeVendor ModelEnd-of-Life
JCU,JCU600TiBDELL SC4020Q4-2021
JCU,QCIF516TiBDDN SFA7990EQ1-2024

Each HPC server has internal SSD storage, as follows:

Server(s) purposeRaw size
Login/Compute nodes480GB (RAID-1) + 7.68TB (RAID-0)
GPU servers240GB (RAID-1) + 960GB (RAID-0)
ESXi servers1600GB (RAID-5)

Note:  Existing GPU and ESXi servers will be decommissioned in 2021.


The following filesystems are configured specifically for JCU research use

FilesystemSizeShared viaDetails
/gpfs01 516TiBGPFSCache for ARDC/RDSI/QCIF approved allocations
/home 512TiBNFSResearchers' home directories
/scratch 80TiBNFSScratch space, shared using NFS
/sw 200GiBNFSSoftware (read-only for researchers)
/fast/tmp 7TiB-Local (to node) SSD scratch space
/tmp300GiB-Local (to node) SSD scratch space

For efficiency across JCU, it is best if your computational research jobs are run under /tmp, /fast/tmp or /scratch.  One completed, only files that are likely to have long-term value should be moved to your home directory or ARDC/RDSI/QCIF approved allocation.


JCU's DELL storage array is very much entry-level - purchased to provide capacity, not performance.

While the DDN storage array has a much higher performance potential, it's performance is limited by the number of disks (64) and SSDs (12) installed in it.

The /gpfs01  filesystem is a medici cache - primary copy of all data is held by QCIF (locations in Brisbane and Springfield).  There is no guarantee of recall time for offline (not currently on JCU cache space) files.  Outages at QCIF, or network issues between JCU and QCIF, will lead to IO errors being seen whenever you try to access/use offline files.  Retrying your task(s) at a later time is what I would suggest.  JCU HPC staff do not have access to fix issues with our medici cache that relate to QCIF(Bne) services.

Your perception of filesystem performance decreases with increasing inodes (file count).  In mid-2019, HPC held over 500,000,000 nodes across 8 filesystems - having less than 10,000,000 inodes per filesystem is desirable.


As of 1-Jan-2020, the following default quotas are configured

FilesystemAccount typeDefault Quotas
/home JCU5TiB & 250,000 inodes
/scratch 

JCU

External

Delegate

5TiB & 1,000,000 inodes

100GiB & 100,000 inodes

100GiB & 100,000 inodes

A research data management strategy is being considered.  Until this strategy is in place, individual quotas may be increased.  Longer term, there may be a merit-approval and/or purchasing process for individual users obtaining an increased quota.

Click for further information about what an inode is

ARDC/RDSI/QCIF Quotas

Many JCU researchers have applied for, and been awarded, an RDSI storage allocation for their research.  There is a default inode quota of 1,000,000 on all allocations.  Higher inode quotas may be configured, however, requestors should consider breaking up their allocation (requesting another allocation) rather than increase inode quota.  History has shown that allocations with high inode counts experience more problems than those with lower inode counts.



Disk Quotainode Quota

Allocation

QCIF(Bne)

JCU CacheJCU Cache
Q003315T5T10,000,000
Q003620T1T1,000,000
Q0042100T55T11,000,000
Q004330T1T1,000,000
Q0044100T20T10,000,000
Q005012T2T1,000,000





Disk Quotainode Quota

Allocation

QCIF(Bne)

JCU CacheJCU Cache
Q011410T2T10,000,000
Q01232T2T1,000,000
Q01245T5T1,000,000
Q012510T2T1,000,000
Q014535T10T2,000,000
Q01481T1T1,000,000
Q014920T5T1,000,000
Q015012T6T1,000,000





Disk Quotainode Quota

Allocation

QCIF(Bne)

JCU CacheJCU Cache
Q01665T5T1,000,000
Q01711T1T1,000,000
Q018445T8T1,000,000
Q018816T6T1,000,000
Q018920T5T1,000,000
Q0195
512G5,000,000
Q01995T5T1,000,000
Q020035T27T20,000,000





Disk Quotainode Quota

Allocation

QCIF(Bne)

JCU CacheJCU Cache
Q020110T5T1,000,000
Q020210T5T1,000,000
Q020310T5T1,000,000
Q0208150T10T1,000,000
Q0210150T40T6,000,000
Q0213200G200G1,000,000
Q021440T40T10,000,000
Q02172T2T5,000,000
Q021950T8T1,000,000
Q022060T60T10,000,000





Disk Quotainode Quota

Allocation

QCIF(Bne)

JCU CacheJCU Cache
Q02228T5T1,000,000
Q023020T5T1,000,000
Q02522T2T1,000,000
Q026160T9T1,000,000
Q0262512G512G1,000,000
Q02661T1T1,000,000
Q026910T5T4,000,000





Disk Quotainode Quota

Allocation

QCIF(Bne)

JCU CacheJCU Cache
Q0308500G500G1,000,000
Q0309500G500G1,000,000
Q03651T1T1,000,000
Q04772T2T1,000,000
Q04784T4T1,000,000
Q06347T7T6,000,000
Q063815T7T1,000,000
Q07505T5T1,000,000
Q111640T10T1,000,000



Q0037 (Wallace2) was a 1PiB RDSI allocation that appeared to be closed (unused) at the time of migration of files to our medici environment.  This combine with problems encountered during said migration, lead to a decision to breakup Q0037 into allocation that would only be housed at QCIF(Bne).  To make the transfer occur in the fastest possible time content on Q0037 was also placed into many large, compressed tarballs.  Given the size and workflows associated with the Wallace2 project, JCU's medici cache would struggle with reintroducing allocations below back into our cache should a researcher request it - extra capacity should be added to our DDN SFA7990E appliance should such a request come through.


Q2024Q2025Q2026Q2027Q2028Q2029Q2030Q2031Q2536
Storage (TiB)15503512052525340
inodes5m10m3m9m10m10m10m10m10m


Contact stefano.montanari1@jcu.edu.au if you need up increase in quota or are interested in obtaining a new allocation.  ARDC/RDSI/QCIF allocations are recommended for any researcher or research group that has a storage requirement in excess of default HPC user quotas.

Other research data stroage options.

HPC provides a location for the primary/trusted copy of your research data only.  Other examples of locations which are responsible locations for your primary/copy of research data are:

  1. Public cloud - e.g., OneDrive and AWS.
  2. Education/Private Cloud - e.g., AARNet CloudStor and ARDC services.
  3. Institutional/Other facilities that offer equivalent or higher level protection mechanisms than JCU HPC.

Personal computing devices (e.g., PCs, USB disks, personal NAS appliance) are definitely NOT a responsible location for the primary/trusted copy of your research data.  From a technical perspective, this is because changes can occur on disks without being noticed and corrected (e.g., search the internet for "silent data corruption").  One of the bigger issues that has been brought to the attention of authorities is the lack of ability to verify/validate research if personal computing devices are involved (e.g., when they house the primary copy of data).


ESSENTIAL UNDERSTANDING (Policy)

All JCU researchers need to read and understand information within https://www.jcu.edu.au/policy/information-and-communications-technology/information-communication-technology-acceptable-use-policy