1. 1 Applied CyberInfrastructure Concepts ISTA 420/520 Fall 2012 1 Nirav Merchant (nirav@email.arizona.edu) Bio Computing & iPlant Collaborative Eric Lyons (ericlyons@email.arizona.edu) Plant Sciences & iPlant Collaborative University of Arizona http://goo.gl/p4j3mhttp://goo.gl/p4j3m or https://sites.google.com/site/appliedciconcepts/ Will Computers Crash Genomics? Science Vol 331 Feb 2011

2. Topic Coverage:  HPC  Racks, Nodes, sockets, chips & cores !  Accessing UA HPC  Resource and Job Management systems 2

3. Docker Vs. Git  Discussion 3

4. What is an HPC  Discussion 4

5. HPC: racks of servers 5

6. HPC: miles of cables 6

7. Look inside… 7 Quad-core, quad-socket Opteron compute node Learn more at: https://computing.llnl.gov/tutorials/linux_clusters/

8. Node, socket, chips/processor, cores 8 For simple english explanation visit http://kb.iu.edu/data/avfb.html

9. Etiquettes for working on HPC  Respect the login node  Understanding the queue and cores+ram limits  Quota  Scratch and disc space  Password less entry via ssh and security ramifications 9

10. Getting started with UA HPC 10 Check course wiki under Getting started..feel free to add new ones !

11. Hands on:  Login to: login.hpc.arizona.edu  Big data: http://snap.stanford.edu/data/web- Movies.html  Defensive data downloading  Find the size of data before downloading  Decide where you want to keep it  Plan space for uncompressed version  Make some accommodations for exploring data, planning 11

12. Hints  Work as a team for this hands on part  http://www.thegeekstuff.com/2012/04/curl-examples/  Estimate the size using curl and “content length”  Quota check  Get space using xdisk  Use alias command to remember (cd to xdisk location by typing cds)  Download using curl  Give me the first 1Mb of data (checksum it for me and count number of userID’s)  zcat and head will come to you rescue 12

13. Git  Create a dir called src/gitfun (hint mkdir –p)  git init  git status try with -s  Create a text file (fun.txt) edit and add text into it  git add fun.txt  git comitt –m “Adding my first file”  git show HEAD^^:fun.txt  git diff HEAD^^:fun.txt./fun.txt 13

14. Homework  You will get email with specifics  I will email you instructions for getting started with Future Grid platforms (https://portal.futuregrid.org/) 14

15. General Architecture 15 CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)

16. Resource & Job Mgmt System 16 CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)

17. Why are you torturing us  This is absolutely boring  I have no use for it  I own my HPC, HTC (and it does what I tell it)  I <3 Amazon and this is never needed on the cloud  Etc etc. 17

18. Resource utilization on HPC systems per parallel job 18 CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)

19. Principal tasks (sequence dia.) 19

20. Scheduling Policy 20

21. Popular Options (commercial and opensource)  SLURM  CONDOR  TORQUE  OAR  SGE  MAUI  MOAB  LSF  PBSPro  LoadLeveler 21

22. Resource and Job Mangement 22 CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)

23. Cloud is no different for RJMS 23

24. Comparison of RJMS 24

25. Some common concepts  Login nodes  Compute nodes  Storage systems  Time: CPU, Wall etc  RJMS systems  Understanding queues  Understanding scheduling  Understanding monitoring  Clever tricks specific to RJMS ! 25

26. PBSpro 26