1. College of Engineering
HCC Workshop
College of Engineering
October 10th, 2014

2. Introduction to LINUX
Operating system like Windows or OS X (but different)
OS used by HCC
Means of communicating with computer
Command Line Interface (instead of GUI)

3. The Terminal Window
After logging in to your account, commands are entered in a terminal window:
You are here
Working in LINUX, you are always located somewhere in your directory tree.
Most common commands: navigating within directory tree, creating, editing, moving and removing files and directories.

4. Getting Started - Logging in
Quick start guide for Windows
Quick start guide for Mac/Linux

5. Your Directories on HCC Computers
Home Directory
- smaller space than work (quota limited by group)
/home/<group>/<userid>
or just
$HOME
Use for:
program binaries, source code, configuration files, etc.
Work Directory
- not backed up
/work/<group>/<userid>
or just
$WORK
Use for:
temporary scratch space (not long-term storage)
I/O for running jobs
Your directories are not shared/mounted across all HCC clusters (ie. Tusker and Crane have separate file systems.)
“print working directory”
“change directory”

6. Other commands to cover
Move/rename files: mv Example: mv oldname newname
Copy files: cp Example: cp myfile myfilecopy
Change the current directory: cd Example: cd mydirectory
See contents of the directory: ls
See directory contents with more details: ls -l
Print working directory: pwd
Print file to screen: more Example: more myfile
Merging multiple files: cat Example: cat file1 file2 > combinedfile
Create empty file: touch Example: touch newfile
Search a string in a file: grep Example: grep mystring myfile
Compress & decompress files: tar, zip, gzip
Remove files: rm Example: rm filetodelete
Remove entire directory: rm -rf Example: rm -rf directorytodelete
See more at:
Exercise:
Using commands on the left: 1. Create a directory named “mydir”.
2. Change into the directory, created a file named “myfile”.
3. Delete “myfile” and “mydir”.

7. Basic exercise - create directory/file

8. One editor : nano or any other you prefer
Start nano by running ‘nano’.
Save: Control + o
Exit: Control + x

9. Exercise - Create submit script
#!/bin/sh
#SBATCH --time=00:05:00
#SBATCH --job-name=helloworld
#SBATCH --error=helloworld.%J.err
#SBATCH --output=helloworld.%J.out
#SBATCH --qos=short
echo “this is my first slurm job”
sleep 30
Using nano, create the text file to the right.
Save it as ‘helloworld.submit’.
We will submit it to SLURM later.

10. Cluster Basics Every job must be run through a scheduler:
Submitting Jobs using SLURM
Software is managed through the module software:
Using Module
Every user has two directories to use:
Where to Put Data

11. SLURM (Simple Linux Utility for Resource Management)
Define the job name
#SBATCH --job-name=MY_Test_Job
s for job notification
#SBATCH
#SBATCH --mail-type=ALL
# (ALL = BEGIN, END, FAIL, REQUEUE)
Requested run time
#SBATCH --time=7-0
# Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds"
Requested number of cores
#SBATCH --ntasks=16 or
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16

12. Request memory for your job
#SBATCH --mem-per-cpu=128
# Set the memory requirements for the job in MB. Your job will be allocated exclusive access to that amount of RAM. In the case it overuses that amount, Slurm will kill it.
Set a pattern for the output file
#SBATCH --output=<filename pattern>
# The filename pattern may contain one or more replacement symbols, which are a percent sign "%" followed by a letter (e.g. %J).
# Supported replacement symbols are: %J Job allocation number. %N Main node name.
Submit an interactive job
Submitting an interactive job is done with the command srun.
$ srun --pty $SHELL
or to allocate 4 cores per node:
$ srun --nodes=1 --ntasks-per-node=4 --mem-per-cpu= pty $SHELL

13. Exercise - Submit hello world job

14. Exercise - Submit MATLAB Job
Invert a large (104 x 104) random square matrix
Using only 1 compute core
Using 10 cores
Compare times
Use a Job Array to invert 5 (or more!) matrices at once

15. Exercise - Submit MATLAB Job
Copy demo files to $WORK
Examine contents of Matlab and Slurm scripts.
Submit first job.
Monitor job.
Edit Slurm submit script (change # tasks) and re-submit.
Compare output files (can also use ‘cat’ or ‘tail’).
Examine submit script for job array and then submit.
Examine job array output.

16. Globus Connect Use Globus Connect to:
Transfer files between HCC clusters (Tusker, Crane, & Sandhills)
Transfer files between your laptop/pc and HCC clusters
Share files with colleagues

17. Globus Connect Getting Started: (HCC-DOCS full instructions)
Sign up for a Globus account
Install Globus Connect Personal on your computer
Activate HCC endpoint(s): hcc#tusker, hcc#crane, hcc#sandhills
Log in to Globus account (online) and start making transfers
Important:
/home is read-only via Globus Connect (transfer from /home, but not to /home).
/work is readable/writable via Globus Connect (can transfer files to and from /work).

18. Exercise: Transfer files with Globus
Option 1: Transfer file with Globus
Download example query file to your laptop: matlab_demo.tar.gz
Transfer file from your laptop to your Crane work directory using Globus
Unpack the tar archive on Crane:
cd $WORK
tar -xzf matlab_demo.tar.gz