1. File Transfer Olivia Irving and Cameron Foss
Blue Waters Student Internship Program 2017

2. Log into SHODOR Windows Users: Mac or Linux Users:
Using putty, connect to
<ENTER>
Mac or Linux Users:
From the terminal,
$ ssh <ENTER>

3. Motivation
Secure and reliable data transfer is essential to a trustworthy network
File transfer is an inherent part of working on a computer.
Data is not stagnant and is often pushed around.
So it is important to be able to work with and manipulate this data in a way that makes it facile and organized .

4. Motivation
Secure and reliable data transfer is essential to a trustworthy network
File transfer is an inherent part of working on a computer.
Data is not stagnant and is often pushed around.
So it is important to be able to work with and manipulate this data in a way that makes it facile and organized .

5. Types of Nodes Gateway (bw and bwbay)
An intermediate node between your system and the login nodes
A ‘head’ node for login nodes
Login (login.shodor.org, h2ologin.ncsa.Illinois.edu)
Nodes where we can compile code, submit batch jobs, or enter an interactive job
For h2ologin there are 12 (or less) login nodes of which there may be many users connected to at any given time.
MOM
An intermediate node between login nodes and compute nodes
A ‘head’ node for compute nodes.
In interactive sessions you are connected to a MOM node.
Compute
where jobs are executed

6. Types of Nodes on Blue Waters
MOM Node
Gateway
End User (You)
Login Node
Compute Node

7. File Transfer Methods Command line (for small files):
Remote Desktop running Linux
(i.e. University Computer)
File Transfer Methods
Command line (for small files):
scp – secure copy protocol
sftp – secure file transfer protocol
rsync – remote synchronization
wget – downloads from URL links
User Interfaces (for large files):
Globus (webpage utility, accessible on any OS)
WinSCP (Windows only)
End User (You)
Cluster, database, server, etc.

8. scp – Secure Copy Copies files between hosts on a network.
Requires authentication via passwords or passphrases
Filenames can be specified with a user and host to transfer between two remote hosts
Local filenames can be made explicitly without user and hostnames

9. scp – Secure Copy BW specific details:
Cannot scp directly to BW because the gateway server is not connected to the filesystem, and cannot ssh (scp?) directly to a login (compute?) node
Training accounts cannot use scp or sftp to "push" data to Blue Waters from a remote host
Can scp from BW once on a login node.
Pure copy and pasting
$ scp local_file

10. rsync – remote file copying tool
fast and versatile
Performs transfers:
To/from a remote host
To/from a remote rysnc daemon
Can also move files locally
Copying and synchronization
Push: $ rsync [options…] file_source host:/remote_file_dest/.
Pull: $ rsync [options…] local_dest
Daemon is a process that runs in the background and performs a specified operation at predefined times or in response to certain events.

11. sftp - Secure File Transfer Protocol
An interactive file transfer program, similar to ftp
Performs operations over an encrypted ssh transport
Uses features of ssh, such as public key authentication and compression
Connects and logs into a remote host and then enters an interactive mode.

12. sftp - Secure File Transfer Protocol
You can also use the ‘ls’ command to list files on the remote system, a feature not present when using scp or rsync.

13. wget - the non-interactive network downloader
free utility for downloading files from the web
supports HTTP, HTTPS, and FTP
HTTP(S) - Hyper Text Transfer Protocol (Secure)
FTP - File Transfer Protocol
non-interacting ⇒ can run in the background

14. For transferring Large Files
Globus
A web interface that is compatible across Operating Systems
Also has a command line interface (CLI)
The preferred file transfer method for Blue Waters
does not support drag and drop
WinSCP
Windows SCP utility
Supports drag and drop
Possible to run on MAC or Linux via Wine (a compatibility layer for running windows applications on MAC or Linux)
Other Cross-Platform Utilities: FileZilla, iMazing (formerly DiskAid), CyberDuck, and many more.
Good old fashion leg work!!
Physically relocating a large set of data may be faster than moving it over the network.

15. Globus Tutorial https://www.globus.org/tags/tutorial
Many, many tutorials
As training accounts do not have RSA tokens, the accounts are not able to use the Blue Waters Globus Online public endpoints.

16. Globus Online
First go to the appropriate tab on the Blue Waters website
then go to globus online.

17. General Outline
To use Globus Online, you must have an account with them. That account is separate from your accounts on the Blue Waters machines and from your account on the Blue Waters user portal, but you can "link" your GO account to your portal account so you can to go Globus Online using your Blue Waters token code and PIN.

18. Globus Online
If you're just creating your Globus Online account, that creation goes as follows:
1.Log into to the Blue Waters portal as usual, with your username and PIN+token code.
2.Go to Globus Online by pulling down the data tab and clicking "Globus Online".
3. You'll authenticate again. This will probably ask for your username and "password". Type in your PIN+token code as before.

19. Globus Online
4. Now connect to your Globus Online Account, and you'll be given the option of creating a Globus Online account. Follow that procedure to create the new account. During this process, select the option to "link" your new Globus Online account to your Blue Waters portal account.
5. Once the Globus Online account is set up, when you click on the Globus Online from within the Blue Waters portal, you can just authenticate with the same information as you log in to the portal with.

20. Adding an Endpoint
Click to get Globus Connect Personal so you can make your own computer an endpoint.

21. Getting Globus Connect Personal
Follow the appropriate directions for your operating system.
& generate your globus connect personal setup key
The key will be asked for once you install the program

22. Globus File Transfer
now enter in an endpoint ncsa#BlueWaters this is what contains your project data and the other endpoint should be from those “Administered by you” This will lead to a local place for your file such as your documents.

23. WinSCP

24. WinSCP
Saved connections

25. WinSCP s Select the File Protocol: SFTP SCP FTP WebDAV
WebDAV – an extension of Hypertext Transfer Protocol (HTTP) that allows clients to perform web content authoring content.
The WebDAV protocol provides a framework for users to create, change and move documents on a server, typically a web server or web share. The most important features of the WebDAV protocol include the maintenance of properties about an author or modification date, namespace management, collections, and overwrite protection. Maintenance of properties includes such things as the creation, removal, and querying of file information. Namespace management deals with the ability to copy and move web pages within a server’s namespace. Collections deal with the creation, removal, and listing of various resources. Lastly, overwrite protection handles aspects related to locking of files.

26. WinSCP s Specify the hostname: h2ologin remote desktops
WebDAV – an extension of Hypertext Transfer Protocol (HTTP) that allows clients to perform web content authoring content.
The WebDAV protocol provides a framework for users to create, change and move documents on a server, typically a web server or web share. The most important features of the WebDAV protocol include the maintenance of properties about an author or modification date, namespace management, collections, and overwrite protection. Maintenance of properties includes such things as the creation, removal, and querying of file information. Namespace management deals with the ability to copy and move web pages within a server’s namespace. Collections deal with the creation, removal, and listing of various resources. Lastly, overwrite protection handles aspects related to locking of files.

27. WinSCP s Your username and password for the specified hostname above.
WebDAV – an extension of Hypertext Transfer Protocol (HTTP) that allows clients to perform web content authoring content.
The WebDAV protocol provides a framework for users to create, change and move documents on a server, typically a web server or web share. The most important features of the WebDAV protocol include the maintenance of properties about an author or modification date, namespace management, collections, and overwrite protection. Maintenance of properties includes such things as the creation, removal, and querying of file information. Namespace management deals with the ability to copy and move web pages within a server’s namespace. Collections deal with the creation, removal, and listing of various resources. Lastly, overwrite protection handles aspects related to locking of files.

28. WinSCP

29. WinSCP
Remote File System
Local File System