1. How to use the HPCC to do stuff
Presentation to QuERG
March 28th, 2016
*Matt’s soothing voice is not a source of necessary nutrients

2. What is HPCC? And iCER? High Performance Computing Center
Collection of computers that has 600 nodes with 7000 computing cores and has large memory nodes (6 TB)
Has lots of available software
iCER is a research unit that maintains MSU’s supercomputer system
Provide 1-on-1 consulting

3. When would I use the HPCC?
Takes too long for computation
Runs out of memory
Needs licensed software
Read/write lots of data

4. How do I connect to the HPCC
You need to set up an account with HPCC
ssh to HPCC
Login to gateway node
ssh to developer node to run test code and submit jobs
Transfer files using SFTP connection

5. How to request a new account
Have your PI fill out the form using the link below  

6. Once you have an account ….
Login using an ssh (Secure Shell) program
Many program options to choose from
MobaXterm
PuTTY
Terminal on Macs
First two can be obtained on portable flash drives from iCER or downloaded
Host Name (or IP address) will be hpcc.msu.edu
Might be better to use rsync.hpcc.msu.edu

7. Login
PuTTY
MobaXterm

8. Logged into gateway
Gateway does not have most of the programs that you want to use. You need to switch to a developer node to access the programs.

9. Gateway nodes Shared drive by anyone with an HPCC account
Only means of accessing the HPCC computing resources
****DO NOT RUN ANYTHING ON THESE NODES!!******

10. Switching to developer mode
Look at the developer Nodes usage
Choose one that has low or med usage
Use ssh to login to switch to developer node
eg: ~]$ ssh dev-intel14
Tells you which node you are logged into and the name of the folder (automatically appears)
~ means Home folder
Code you type to switch to dev-intel14

11. Begin working The command line takes unix commands to do work
Can use many different text editors to edit files
emacs file.ext
nano file.ext
vi file.ext
joe file.ext
cat file.ext (This only prints what is in the file cannot edit)

12. Basic Linux Commands Command Meaning cd directory Change Directory
cd Down one directory
cd ../ Down two directories
cd Return to previous Directory
cd ~ Go to home/username
mkdir directory Make named directory
rmdir directory Remove an empty directory

13. Basic Linux Commands Command Meaning
ls Show contents of current folder
Some options for ls command
-a list all files and directories
-F append indicator (one of to entries
-h print sized
-l list with a long listing format
-t sort by modification time

14. More Linux commands Command Meaning cp source destination Copy files
cp –r source destination Copy files recursively: files and directories
mv source destination move a file (can be used as a rename command!)
pwd Show current path
rm filename.ext remove file
rm –r folder/ remove directory recursively (i.e. including all subdirs and files)

15. How to find commands? How to find how to do something?
Google is usually the first place to look
An exhaustive list
• A useful cheatsheet
• Explain a command given to you

16. MSU HPCC specific commands
Command Meaning
sj show jobs
gmod show the home screen with development node use levels
qsub submit a job to the scheduler
getexample shows a list of available examples that can be loaded to current directory

17. Module This is how you access different softwares
module list : prints list of the loaded programs
module load OpenBUGS : load OpenBUGS
module unload modulename : unload a module
module spider keyword search the modules for a key word and lists what can be loaded includes the different versions
module purge : Unload all modules
R is automatically loaded. But can use this to load a certain version of R if needed
module load JAGS load JAGS package
can load jags but was unable to load rjags so won’t use it in presentation. Someone could try loading the rjags library to their personal directory if needed
should be somewhere in this link

18. Working on the HPCC Can run small tasks on the Developer nodes
If runs for longer than 2 hours or uses too much memory it can be canceled without warning
Use Developer Nodes to test your code and determine the resources you need to run jobs
When it takes a long time to run you will then submit jobs to the scheduler
qsub myjob.sub

19. What do I mean by a Job?
A job is just a list of commands that I tell the computers to do in a job script
It also contains the requirements to run the job
This will then be submitted to the scheduler and eventually run on the computers when resources become available based on what you request
When the job is done running you can check your results by looking at the files (etc.) created

20. How the scheduler works
Ranks submitted jobs based on a priority system
Priority is influenced by how long they have been in the queue and resources they request
Large jobs may be in the queue for a long time before they begin running
Jobs less than 4 hours typically start running the fastest (sometimes immediately)

21. Creating job scripts What is needed in a job script?
List of required resources
Run time
Memory
Number of nodes
number of cores per node
All command line instructions needed to run the computations

22. Typical submission script
Special system command
Login to Shell
Resource requests
Shell command
Special Environment variables
Shell command
Shell command
Special Environment variables
Instructions to scheduler

23. Job Script details # is normally a comment except
#! Special system commands
#!/bin/bash –login this logs you so you can run the job
#PBS instructions to the scheduler
#PBS –l nodes=n,ppn=p
#PBS –l walltime=hh:mm:ss
#PBS –l mem=2GB (!!not per core but total memory)

24. Instructions to scheduler
All lines starting with #PBS need to be above the first non-commented line in the script. If they are below the first non-commented line of code, the scheduler will not read them, leading to unexpected behavior.
More options at
All jobs must have
Walltime requesting
Memory requested
# of nodes requested and processers per node (ppn)

25. Advanced Environment Variables
PBS_JOBID
the job number of the current job
PBS_O_WORKDIR
The working directory from which the job was submitted
${} tells the computer that this is a variable
e.g. mkdir ${PBS_O_WORKDIR}/${PBS_JOBID}

26. Ways to run R Rscript myRprogram.r R < myRprogram.r --no-save
Does not save workspace
Or R < myRprogram.r --save
Save workspace
Or R < myRprogram.r --vanilla
do not read any user or site profiles or restored data at start up and to not save data files at exit
R CMD BATCH myRprogram.r

27. Time Submitting jobs Queue Run qsub submission.script
Submit a job to the queue will return a job ID#
Typically looks like cmgr0
Time to completion
Queue
Run
Time

28. Checking on your jobs qdel jobID# showq –u userid sj checkjob jobID#
delete a job from the queue
showq –u userid
show the current job queue of the users sj
show the status of jobs (running, eligible, or blocked)
checkjob jobID#
Check status of the job
showstart –e all jobID#
Show the estimated start time of the job

29. Scheduling tips
Requesting more resources does not make a job run faster unless running a parallel program
more resources requested makes it “harder” for the schedule to reserve those resources
First time: over-estimate how many resources you need, and then modify appropriately
qstat –f ${PBS_JOBID} put this code at the bottom of the script to show you resources used information when the job is done

30. Advanced Scheduling Tips
Large portion of clusters are buy-in that can only run jobs that are less than 4 hours
Most nodes have at least 24GB memory
Half have at least 64 GB of memory
Few have more than 64Gb of memory
(i.e. harder to schedule jobs that requests lots of memory)

31. System Limitations
10 eligible jobs in the queue (other will be temporarily blocked until jobs start running)
520 running cores (nodes*ppn)
1000 submitted jobs
1 week of walltime
ppn=64
2TB memory on a single core
~200 GB Hard Drive

32. Job Completion
By default the job will automatically gnerate two files when it completes:
Standard Output: E.g. jobname.o
Standard Error E.g. jobname.e
You can combine these files if you add the join option in our submission script: #PBS –j oe
You can change the output file name
#PBS –o /mnt/scratch/home/netid/myoutputfile.txt

33. Transferring Files SFTP program (Secure File Transfer) Two options are
MobaXterm
WinSCP
Gateway is : rsync.hpcc.msu.edu
Drag and drop files between personal computer and HPCC

34. Where to go for help iCER office hours Monday and Thursdays 1 to 2
Biomedical & Physical Sciences Building 567 Wilson Road, Room 1440
Also by appointment
: contact HPCC by submitting a ticket/contact form (msu login)
wiki.hpcc.msu.edu :HPCC User Wiki
icer.msu.edu : iCER Home
hpcc.msu.edu :HPCC Home

35. How to convert windows files to unix
dos2unix
Converts special end-of-line characters from windows format to unix
dos2unix myRprogram.r
Only necessary if you use an editor that does not use unix end of line character