IMPORTANT POINTS:
- Officially, JCU's OneDrive for Business platform hasn't been provisioned for housing research data, however JCU staff/students may choose to place their research data on OneDrive.
- OneDrive software installed on HPC is NOT provided or supported by Microsoft. Technology Solutions staff do not use routinely use OneDrive on HPC - two of us have successfully run the commands below once.
- HPC storage does NOT exist to serve as a sync client to your OneDrive files. The OneDrive software on HPC should only be used to upload files on HPC to OneDrive - e.g., for data protection or archiving (long-term).
- A long-term storage option for JCU research data is being considered. AARNet CloudStor is the most likely platform at this time. HPC cluster storage may be "scratch space" only, possibly with routine deletion of "old" files configured.
Background:
Extensive documentation on the linux onedrive connector capabilities can be found at https://github.com/abraunegg/onedrive/blob/master/docs/USAGE.md - the information provided below is a very brief introduction for getting started. At the time of writing this document, HPC staff had only completed very limited testing of the Linux interface to OneDrive. The onedrive command can also be used to connect to SharePoint libraries (untested).doesn't have the capabilities provided by Windows/MacOS OneDrive clients, but it allows you to copy files housed on HPC to OneDrive.
Brief Documentation
A command line based interface to OneDrive has been installed on HPC cluster servers. There is some configuration required before you start using the command(s) provided.
Create a "onedrive" directory somewhere within your scratch space directory.
No Format mkdir -p -m 700 /scratch/jc012345`whoami`/onedrive
Modify your environment by loading the onedrive module.
No Format module load onedrive
Authenticate with OneDrive:
Issue the command
No Format onedrive
You should get a response something like:
No Format [jc012345@ln01 ~]$ onedrive Authorize this app visiting: https://..... Enter the response uri:
- Copy and paste the URL into a browser (on your desktop/laptop computer) and hit the Enter or Return key to load content. You will then need to authenticate to OneDrive - for most JCU staff/students SSO (single sign on) should take care of it for you. Once authentication has been successful, a blank page should be returned (may change in future). You need to copy the URI/URL that was returned (where you pasted the URL earlier) and paste the contents as a response to the
Enter the response uri:prompt. You should receive a message that indicates successful authentication.
Now it's time to configure your use of
onedriveon HPC. The following is very similar to a configuration that has been tested by HPC staff for use of OneDrive as an archive for files on HPC.No Format mkdir -p ~/.config/onedrive cat <<EOF >~/.config/onedrive/config sync_dir = "/scratch/jc012345`whoami`/onedrive" skip_file = "~*|.~*|*.tmp" skip_dotfiles = "true" skip_symlinks = "true" upload_only = "true" no_remote_delete = "true" remove_source_files = "false" EOF
To confirm your configuration (may be larger than the contents of your configuration file), execute
No Format onedrive --display-config
- You are now ready to upload files to or download files from OneDrive, remembering that HPC storage is for research files only - all other files should be moved to another location (removed from HPC). For uploading files to OneDrive, copy/move files to the "
sync_dir" you configured - e.g., to move all *.dat files in your current working directory, usemv *.dat /scratch/jc012345`whoami`/onedrive. Two aliases have been created for upload/download convenience. To upload files from your "
sync_dir", simply use the command:No Format onedrive-upload
- The progress of uploads will show up on your terminal. The command should tell you whether transfers were successful or unsuccessful. When finished, you may choose to verify success accessing your OneDrive storage from your desktop/laptop computer.
...
At this stage, it looks like download capabilities are tied to a full sync'ing everything from OneDrive. Please be aware that files Files you have uploaded to OneDrive from another personal/work computing device may not be permitted on HPC . If a file (research files only). Therefore, you should not be sync'ing (on HPC servers) data with OneDrive. Point in time downloads from OneDrive to HPC can be done, providing you delete all content that isn't directly related to your research activities, it must not be placed on HPC infrastructure. The most likely command you will use is:
...
...
onedrive --synchronize --download-onlyAccess to Teams/Sharepoint Libraries
The linux onedrive client installed on HPC systems is capable of working with Teams/Sharepoint libraries. This feature should only be used if the Team you are connecting to is housing research data only. Such synchronization should only be synced to group shares (not home directories) on HPC. You can list drive IDs of a sharepoint library using:
| No Format |
|---|
onedrive --get-O365-drive-id '<insert team/library name here>' |
However, due to JCU's O365 storage limit, files on HPC should not be sync'ed with Teams/Sharepoint document libraries. Such libraries can be synced to other computing devices (e.g., your desktop computer).
HPC filesystems
The future architecture of the JCU HPC cluster is under review - based on a government & university drive to utilize 'cloud' infrastructure where practical & sustainable. Technology solutions (including HPC) staff are investigation options for a future HPC cluster where >=95% of storage is likely to be scratch space, with routine deletion of old files. The
| No Format |
|---|
no_remote_delete = "true" |
configuration entry (in ~/.config/onedrive/config) will be essential if this future state becomes reality.