1. High Performance Computing Systems for IU Researchers – An Introduction IUB Wells Library 10-Sep-2012 Jenett Tillotson jtillots@iu.edu George Turner turnerg@iu.eduturnerg@iu.edu High Performance – Systems hps-admin@iu.edu

2. Getting Started BigRed:http://kb.iu.edu/data/avjx.htmlhttp://kb.iu.edu/data/avjx.html Quarry:http://kb.iu.edu/data/avkx.htmlhttp://kb.iu.edu/data/avkx.html

3. Getting an account http://itaccounts.iu.edu/ Manage my IU computing accounts create more accounts choose "BigRed" or "Quarry” Takes about 15 minutes You will receive a welcome email

4. Logging in ssh bigred.teragrid.iu.edu ssh quarry.teragrid.iu.edu MOTD

5. Nodes BigRed: b1-b1050, s1c1b1-s19c3b14 Quarry-classic: b001-b140, q001-q140 Quarry-pg: p1-p230, pg1-pg230

6. Head nodes BigRed: b513 - b516 Quarry-classic: b001 - b004 Quarry-pg; p230

7. Wild West nodes BigRed: b509-b512 Quarry: q005-q008

8. Compute nodes BigRed: b1-b508, b561-b1050 Quarry: q009-q140, pg1-pg229

9. File systems Home directory DataCapacitor (Lustre)

10. File systems : Home directory /N/u/{username}/BigRed /N/u/{username}/Quarry ${HOME} 10GB quota quota -v Slow, limited, backed up

11. File systems : DataCapacitor (lustre) http://kb.iu.edu/data/avvh.html IU users: /N/dc/scratch/{username} Fast, unlimited, not backed up Permanent project file space available

12. Softenv Modifies environment ${PATH} and ${MANPATH} softlist soft add.soft resoft

13. Resource Manager Keeps tracks of resources: nodes, jobs, queues BigRed – LoadLeveler llsubmit llq llcancel llclass Quarry – TORQUE qsub qstat –a –u $USER qdel qstat -Q

14. Queues : BigRed LONG: 32 nodes/job, 64 nodes/user, 14 days wall clock time NORMAL: 256 nodes/job, 512 nodes/user, 2 days wall clock time SERIAL: 1 proc/job, 512 proc/user, 2 days wall clock time DEBUG: 4 nodes/jobs, 4 nodes/user, 15 minutes wall clock time, 1 idle job Limit of 768 jobs in all the queues per user Limit of 16 idle jobs per user (except for the DEBUG queue)

15. Queues : Quarry long: 42 nodes/job, 14 days wall clock time, 50 jobs/user normal: 6 nodes/job, 7 days wall clock time, 500 jobs/user serial: 1 node/job, 12 hours wall clock time, 500 jobs/user debug: 4 nodes/job, 15 minutes wall clock time,2 jobs/user himem: 28 nodes/job, 14 days wall clock time, 50 jobs/user batch: default queue Limit of 16 idle jobs per user

16. Job Scripts http://kb.iu.edu/data/axpz.html

17. BigRed #!/bin/bash -l # @ step_name = step1 # @ initialdir = /N/u/jtillots/BigRed/myoutput # @ output = step1.out # @ error = step1.err # @ notification = always # @ notify_user = jtillots@indiana.edu # @ class = DEBUG # @ wall_clock_limit = 15:00 # @ account_no = NONE # @ queue /bin/date sleep 10 /bin/date

18. Quarry #!/bin/bash -l #PBS -N step1 #PBS -j oe #PBS -k o #PBS -m abe #PBS -M jtillots@indiana.edu #PBS -q debug #PBS -l nodes=1,walltime=15:00 /bin/date sleep 10 /bin/date

19. Job ids BigRed: s10c2b5.{jobid}.0 Quarry: {jobid}.qm2

20. Scheduler : Moab Decides which jobs get run on what nodes at what time. showq by queue:-w class={queuename} running jobs:-r idle jobs:-i blocked jobs:-b checknode checkjob showstart Single-user mode

21. Job Priority mdiag -p Fair share XFactor QOS Backfill showbf