ISG We build general capability Purpose After this tutorial, you should: Be comfortable submitting work to the batch queuing system of olympus and be familiar with basic terms Be able to monitor the progress of your submitted jobs Understand how to efficiently use the filesystem on olympus Be able to modify the supplied scripts for your work Know where to go for help! 1

ISG We build general capability Types of Jobs Serial Jobs: individual, independent jobs that run using a single core of a single processor on a single node Multicore Parallel Jobs: use multiple cores on a single node OpenMP Message Passing Jobs: can use multiple cores distributed over multiple nodes Openmpi 2

ISG We build general capability Job Scripts A job script is a step-by-step recipe for completing work on a compute cluster The recipe is written in a scripting language; we will use bash in our examples In order to execute the instructions on the Olympus compute cluster, we will use the qsub command 3 Examples can be found on the olympus gitlab site:

ISG We build general capability What does this look like in a job submission script? 4 local_scratch_path="/net/$execution_compute_node$LOCAL“ # make a directory for this job; name created using job id local_working_dir_name="$PBS_JOBID.output.directory" local_working_dir_net_path="$ local_scratch_path /$local_working_dir_name" # create the directory mkdir -p $local_working_dir_net_path # dump all environment variables to a compressed file env | gzip > $local_working_dir_net_path/$PBS_JOBID.env.gzip # create a symlink to the local working dir, available through the execution # compute nodes NFS export ln -s $local_working_dir_net_path $PBS_O_WORKDIR/$local_working_dir_name Set an environment variable defining the path to the “local” directory. Define a directory name that is unique to your job. Make that directory. Make the output of your job go to that directory Create a shortcut to your output so you can access it on the head node.

ISG We build general capability PBS Directives in a Job Submission Script 5 #!/bin/bash –f # Remarks: A line beginning with # is a comment. # A line beginning with #PBS is a PBS directive. # PBS directives must come first; any directives after the first executable statement are ignored. #PBS -N test.bash # #PBS -o stdout_file # #PBS -e stderr_file The “hash-bang” or “shebang” specifies the scripting language used An active PBS directive Commented-out PBS directives

ISG We build general capability Submitting a Job and Monitoring Progress After submitting your script with qsub it will be entered into the queue A queue is a prioritized list of jobs to be completed Once submitted, the status of your job can be viewed with qstat “qstat –a” gives you more verbose output After your job completes the output of your job will be available in your home directory 6

ISG We build general capability Example Jobs Clone the git repository with the command: “git clone Enter the examples directory: “cd olympus.git/examples” View the directories by typing “ls”, you should see: “sanity”: a basic diagnostic sanity check “mpihello”: a simple example of a parallel multinode mpi code “flute”: a basic, real-world example of parallel mpi code “fred”: a basic, real-world example of OpenMP multithreaded code 7

ISG We build general capability Example: “sanity” Go to the examples/sanity directory View the contents using “less”: “less sanity.bash” Navigate with up and down arrows, exit by pressing ‘q’ The script is heavily commented, explaining each step Submit your job! “qsub sanity.bash” 8 Can also view here:

ISG We build general capability Using Olympus File Systems Each node in Olympus has a “local” disk, physically located inside the node. Fast, reliable for work on its own node Olympus has a “shared” file system that is accessible to all nodes via the network. This is where your home directory is While running, jobs should write all of its files to the “local” disks. For convenience, the “local” directory can be accessed on the head node through the path /net/ /tmp Example: if you want to go to the node n002’s local disk, it the path would be /net/n002/tmp. 9

ISG We build general capability Try the other examples Navigate to the other diectories (flute, fred, mpihello) Each contains a “README” text file with instructions for submission Each job should take only a few minutes 10