1 Applied CyberInfrastructure Concepts ISTA 420/520 Fall Nirav Merchant Bio Computing & iPlant Collaborative Eric Lyons Plant Sciences & iPlant Collaborative University of Arizona or Will Computers Crash Genomics? Science Vol 331 Feb 2011

Topic Coverage:  HPC  Racks, Nodes, sockets, chips & cores !  Accessing UA HPC  Resource and Job Management systems 2

Docker Vs. Git  Discussion 3

What is an HPC  Discussion 4

HPC: racks of servers 5

HPC: miles of cables 6

Look inside… 7 Quad-core, quad-socket Opteron compute node Learn more at:

Node, socket, chips/processor, cores 8 For simple english explanation visit

Etiquettes for working on HPC  Respect the login node  Understanding the queue and cores+ram limits  Quota  Scratch and disc space  Password less entry via ssh and security ramifications 9

Getting started with UA HPC 10 Check course wiki under Getting started..feel free to add new ones !

Hands on:  Login to: login.hpc.arizona.edu  Big data: Movies.html  Defensive data downloading  Find the size of data before downloading  Decide where you want to keep it  Plan space for uncompressed version  Make some accommodations for exploring data, planning 11

Hints  Work as a team for this hands on part   Estimate the size using curl and “content length”  Quota check  Get space using xdisk  Use alias command to remember (cd to xdisk location by typing cds)  Download using curl  Give me the first 1Mb of data (checksum it for me and count number of userID’s)  zcat and head will come to you rescue 12

Git  Create a dir called src/gitfun (hint mkdir –p)  git init  git status try with -s  Create a text file (fun.txt) edit and add text into it  git add fun.txt  git comitt –m “Adding my first file”  git show HEAD^^:fun.txt  git diff HEAD^^:fun.txt./fun.txt 13

Homework  You will get with specifics  I will you instructions for getting started with Future Grid platforms ( 14

General Architecture 15 CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)

Resource & Job Mgmt System 16 CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)

Why are you torturing us  This is absolutely boring  I have no use for it  I own my HPC, HTC (and it does what I tell it)  I <3 Amazon and this is never needed on the cloud  Etc etc. 17

Resource utilization on HPC systems per parallel job 18 CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)

Principal tasks (sequence dia.) 19

Scheduling Policy 20

Popular Options (commercial and opensource)  SLURM  CONDOR  TORQUE  OAR  SGE  MAUI  MOAB  LSF  PBSPro  LoadLeveler 21

Resource and Job Mangement 22 CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)

Cloud is no different for RJMS 23

Comparison of RJMS 24

Some common concepts  Login nodes  Compute nodes  Storage systems  Time: CPU, Wall etc  RJMS systems  Understanding queues  Understanding scheduling  Understanding monitoring  Clever tricks specific to RJMS ! 25

PBSpro 26