Zellescher Weg 12 Trefftz-Building – HRSK/151 Phone Guido Juckeland Center for Information Services and High Performance Computing (ZIH) Introduction to High Performance Computing at ZIH The LSF Batch System

Slide 2 - Guido Juckeland Agenda What is a batch system? Batch queues on the Altix/Deimos Host groups on Deimos Starting, stopping, and monitoring batch jobs Batch scripts

Slide 3 - Guido Juckeland What is a batch system?

Slide 4 - Guido Juckeland Concept of a batch system Login- Host - Compile - File transfers Master- Host Compute- Host... Submission of the batch job Execution of the batch job

Slide 5 - Guido Juckeland Deimos Batchsystem Login- Host - Compile - File transfer Master- Host p1d001 p1d002 p1d003 p2s Submission of the batch job Execution of the batch job Login- Host

Slide 6 - Guido Juckeland Mars Altix Batchsystem Login- Host - Compile - File transfer Master- Host Jupiter Saturn Uranus Mars Submission of the batch job Execution of the batch job

Slide 7 - Guido Juckeland What is a job? A piece of work (e.g. a script, command, or application call) that was handed over to the control of the batch system (using „bsub“) The batch system determines: –Execution time of the job (when?) –Placement of the job on the hosts (where?) –If needed the interupt of the job The batch system also does the accounting for jobs with respect to the available project CPU time. Special case: Interactive Jobs

Slide 8 - Guido Juckeland A job‘s life cycle

Slide 9 - Guido Juckeland What is a host? Deimos/Phobos: –Login node –Compute node Altix: –Alle 4 Partitionen A host contains a number of job slots (number of sequential jobs that could be placed on a host): –Deimos: 1 node -> 2-8 job slots (since 2-8 CPUs) –Phobos: 1 node -> 2 job slots (since 2 CPUs) –Altix: Mars -> 346 job slots (since 346 available CPUs) Jupiter, Saturn, Uranus -> 506 job slots (since 506 available CPUs)

Slide 10 - Guido Juckeland What is a queue? Queue = Alignment of things/people waiting for some event Batch queue = Queue for batch jobs Usually different queues have different limits with respect to max. run time or max. avail job slots One batch queue for the whole system –Advantage: Easy to administer –Disadvantage: User has to specify all job parameters (CPU time, memory usage,…) Multiple batch queues for the whole system (Altix, PC farm) –Advantage: Easy to use for the user –Disadvantage: More difficult to manage

Slide 11 - Guido Juckeland Batch queues on the Altix/Deimos

Slide 12 - Guido Juckeland Batch queues on Mars NameUsersCPUs Time limit default /max Hosts interactiveAll h / 12 hAll ilrILR h /24 hAll smallAll h / 5 dAll intermediateAll64 – h / 5 dAll largeAll256 – h / 24 hAll

Slide 13 - Guido Juckeland Batch queues on Deimos NameUsersCPUs Time limit default /max Hosts interactiveAll h / 12 hAll smallAll h / 5 dAll rtcSelected users h All but fat_quads intermediateAll9 – 12712h / 5dAll largeAll128 – h / 24 hAll nightexpressMPI_CBG14 hAll gaussGauss users hfat_quads

Slide 14 - Guido Juckeland Host groups on Deimos

Slide 15 - Guido Juckeland Host groups on Deimos p1_hosts - Phase 1 nodes p2_hosts - Phase 2 nodes single_hosts - Single CPU nodes dual_hosts - Dual CPU nodes quad_hosts - 16 GByte Quad CPU nodes fat_quads - 32 GByte Quad CPU nodes single{1|2}_hosts - Phase 1/2 Single CPU nodes dual{1|2}_hosts - Phase 1/2 Dual CPU nodes quad{1|2}_hosts - Phase 1/2 16 GByte Quad nodes express_hosts - Knoten für Queue nightexpress

Slide 16 - Guido Juckeland Starting, stopping, and monitoring batch jobs

Slide 17 - Guido Juckeland Starting a batch job (1) Command: bsub Call with: bsub -n [parameter] [Command parameters] Parameters: – -n : Number of job slots to use (CPUs) – -q : Selects a specific queue for the job – -W : Max. runtime of the job (format: H:MM) – -e : Redirects all error output (stderr) to „file“ – -o : Redirects all standard output (stdout) to „file“ – -M : Max. amount of main memory needed by the job – -m : Specifies a certain host (group) for the job execution – -x : Uses the execution host exclusively by the job – -Is: Interakte job ( bsub -Is bash -l)

Slide 18 - Guido Juckeland Starting a batch job (2) – -w done( ): Start job when job with ID „job-id“ is done successfully Example: bsub pwd Job is submitted to default queue. Sequential program: –Altix/Phobos/Deimos/SX-6: bsub./a.out MPI-parallel program: –Altix: bsub -n pamrun./a.out –Deimos: bsub -n -a openmpi mpirun.lsf./a.out OpenMP-parallel / Multithreaded program: –Altix: bsub -R "span[hosts=1]" -n./a.out –Deimos: bsub -R "span[hosts=1]" -n {1-8}./a.out

Slide 19 - Guido Juckeland Modifying a batch job Command: bmod Attention! Works usually only with pending jobs (PEND) Call with: bmod [Parameter] Parameter: – -n : Number of job slots to use (CPUs) – -q : Selects a specific queue for the job – -W : Max. runtime of the job (format: H:MM) – -e : Redirects all error output (stderr) to „file“ – -o : Redirects all standard output (stdout) to „file“ – -M : Max. amount of main memory needed by the job – -m : Specifies a certain host (group) for the job execution – -x: Uses the execution host exclusively by the job

Slide 20 - Guido Juckeland Cancelling a batch job Command: bkill Call with: bkill Special cases: –bkill 0 cancels all jobs of a user –bkill sends only a SIGKILL to the application -> if the process executed by the job does not respond to that signal, LSF cannot abort the job

Slide 21 - Guido Juckeland Modification of the order of pending jobs Default: Execution order = Order of arrival Commands: btop/bbot Call with: btop/bbot Example: bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME juckel PEND express host pwd Jun 15 10: juckel PEND express host pwd Jun 15 10: juckel PEND express host pwd Jun 15 10:33 bbot Job has been moved to position 1 from bottom. bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME juckel PEND express host pwd Jun 15 10: juckel PEND express host pwd Jun 15 10: juckel PEND express host pwd Jun 15 10:33

Slide 22 - Guido Juckeland Monitoring a job status (1) Command: bjobs Call with: bjobs [Parameter] [Job-ID] Parameters: –With all: Shows a list of all jobs of the user with [Job-ID] –-p [Job-ID]: Shows the reason why the job is pending –-l [Job-ID]: Shows detailed information about the job with [Job-ID] –-q : Shows all the users jobs in the queue –-r : Show only jobs in status (RUN) Example: bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME juckel PEND express host pwd Jun 15 10: juckel PEND express host pwd Jun 15 10: juckel PEND express host pwd Jun 15 10:33

Slide 23 - Guido Juckeland Monitoring a job status (2) Possible states of a job: –PEND (pending): Job is waiting to be executed –RUN (running): Job is currently executed –UNKN (unknown): Satus of the job is unknown (usually happens when the job ran on a node that crashed) –SUSP (suspended): Job was stopped

Slide 24 - Guido Juckeland Information about a job after it has finished Command: bhist Call with: bhist -l Example: bhist -l Job, User, Project, Command Thu Jun 15 10:33:26: Submitted from host, to Queue, CWD ; Thu Jun 15 10:34:06: Job moved to position 1 relative to by user or ad ministrator ; Thu Jun 15 10:39:19: Dispatched to ; Thu Jun 15 10:39:19: Starting (Pid 23143); Thu Jun 15 10:39:19: Running with execution home, Execution CWD, Execution Pid ; Thu Jun 15 10:39:19: Done successfully. The CPU time used is 0.0 seconds; Thu Jun 15 10:39:19: Post job process done successfully; Summary of time in seconds spent in various states by Thu Jun 15 10:39:19 PEND PSUSP RUN USUSP SSUSP UNKWN TOTAL

Slide 25 - Guido Juckeland Displaying all jobs in the system (1) Command: qstat -a Example: qstat -a... small; type=BATCH; [ENABLED]; pri=20 10 run; 0 wait; REQUEST NAME REQUEST ID USER STATE 1: CRANp_S2_PBE rbarthel RUNNING 2: ts_rekom_eta drees RUNNING 3: tricomplex_ drees RUNNING 4: BCPPp rbarthel RUNNING 5: iC_C20H9p rbarthel RUNNING 6: iA_C21OH10p rbarthel RUNNING 7: iC_C20H rbarthel RUNNING 8: iA_C20H rbarthel RUNNING

Slide 26 - Guido Juckeland Displaying all jobs in the system(2) Command: lsf_info (developed at ZIH) Example: lsf_info Running Jobs JOBID USER PROJECT QUEUE PROC START TIME WALL TIME USED UTILIZATION mlieber ozon large Jun 10:43 0:01 of 0:15 44% rbarthel nano1 small 1 13.Jun 05:15 2T/ 5h of 2T/23h 99% drees akat small 2 13.Jun 11:43 1T/23h of 3T/ 0h 43% drees akat small 2 14.Jun 16:35 18:09 of 3T/ 0h 11% rbarthel nano1 small 1 14.Jun 21:07 13:37 of 2T/23h 97%... Pending Jobs JOBID USER PROJECT QUEUE SUBMITTED PROC muel zhr stresstest 28.Mai 11: muel zhr stresstest 28.Mai 11:

Slide 27 - Guido Juckeland System status Command: nodestat (developed at ZIH) Call with: nodestat [Hostgroup] Example: deimos101:~ # nodestat nodes available: 724/724 nodes damaged: jobs running: 1576 | cores closed (exclusive jobs):118 jobs wait: 909 | cores closed by ADMIN:20 jobs suspend: 0 | cores working:2300 jobs damaged: 0 | normal working cores: 2576 cores free for jobs:

Slide 28 - Guido Juckeland Displaying the status of the batch queues (1) Command: bqueues Call with: bqueues [Parameter] [Queue name] Parameters: –without: Shows a summary of all queues or the specified queue –-l [Queuename]: Shows detailed information about all queues or the specified queues Example: bqueues QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP interactive 30 Open:Active large 25 Open:Active gauss 20 Open:Active small_long 20 Open:Active small 20 Open:Active

Slide 29 - Guido Juckeland Displaying the status of the batch queues(2) Possible states of the queues: –Active: Queue accepts and executes jobs –Inactive: Queue accepts jobs but execution is delayed –Closed: Queue does not accept jobs and jobs are not executed

Slide 30 - Guido Juckeland Information about the compute hosts Kommando: bhosts Call with: bhosts (Altix), bhosts batch_hosts (Phobos,Deimos) Beispiel: bhosts HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV merkur ok venus ok Possible states of hosts: –OK: Host accepts jobs –Closed: Host does not accept jobs (host is either full or closed by the admin) –Unkown: Host is crashed

Slide 31 - Guido Juckeland Batch scripts

Slide 32 - Guido Juckeland Layout #!/bin/sh #BSUB -n 4 #BSUB -q small #BSUB -a openmpi mpirun.lsf./a.out

Slide 33 - Guido Juckeland Submission of a job with a batch script Command: bsub Call with: bsub [Parameters] Example: bsub < test.sh Job is submitted to queue.

You need help later on? There are no stupid questions or requests!! Central drop off point: Central information point: Read our mail-announcements: Discuss your problem with other ZIH HPC users: Slide 34 - Guido Juckeland