Quick Tutorial on MPICH for NIC-Cluster CS 5802 Class Notes

NIC CLUSTER OVERVIEW Start page: http://hpc.mst.edu/ Node Allocation and Usage policy: http://hpc.mst.edu/accessandpolicies/ The Shared NIC Cluster Hardware and Software The NIC cluster had 64-bit nodes (http://hpc.mst.edu/about/hardware/) connected with an Ethernet network. READ http://hpc.mst.edu/quickstart/ To get started.

PBS Job Scripts NIC cluster uses PBS (Portable Batch System) Why? Improves overall system efficiency Fair access to all users since it maintains a scheduling policy Provides protection against dead nodes

How PBS works User writes a batch script for the job and submits it to PBS with the qsub command. PBS places the job into a queue based on its resource requests and runs the job when those resources become available. The job runs until it either completes or exceeds one of its resource request limits. PBS copies the job’s output into the directory from which the job was submitted and optionally notifies the user via email that the job has ended.

Step 1: Login Off-campus machine: Connect to campus using MST VPN > ssh nic.mst.edu On-campus machine (VPN is not required) Use the following to set up your MPI path correctly. $ module load openmpi/gnu To make this your default run $ savemodules Visit http://hpc.mst.edu/examples/openmpi/c/ to get information about OpenMP

No DFS Support DFS files are not directly accessible at the cluster Use sftp command to transfer any files from your DFS space (S: drive) e.g. > sftp minersftp.mst.edu > get X.c > quit You may also use WinSCP in Windows.

Step 2: Compile MPICH Programs Syntax: C : mpicc –o hello hello.c C++ : mpiCC –o hello hello.cpp Note: Before compilation, make sure the MPICH library path is set or use the export command like below: export PATH=/opt/mpich/gnu/bin: $PATH Executable file

Step 3: Write PBS batch script file Ex1: A simple script file (pbs_script) A job named “HELLO” requests 8 nodes and at most 15 minutes of runtime. #!/bin/bash #PBS –N HELLO #PBS –l walltime=0:15:00 #PBS –l nodes=8 #PBS –q @nic-cluster.mst.edu mpirun –n 8 /nethome/users/ercal/MPI/hello

Some PBS Directive options -N jobname (name the job “jobname”) -q @nic-cluster.mst.edu (The cluster address to send the job to) -e errfile (redirect standard error to a file named errfile) -o outfile (redirect standard output to a file named outfile) -j oe (combine standard output and standard error) -l walltime=N (request a walltime of N in the form hh:mm:ss) -l cput=N (request N sec of CPU time; or in the form hh:mm:ss) -l mem=N[KMG][BW] (request total N kilo| mega| giga} {bytes| words} of memory on all requested processors together) -l nodes=N:ppn=M (request N nodes with M processors per node)

Step 3.1: Submit a Job Use PBS command qsub Syntax : qsub pbs-job-filename Example : > qsub pbs_script returns the message 555.nic-p1.srv.mst.edu (555 is the job ID that PBS automatically assigns to your job)

Result after job completion An error file and an output file are created. The names are usually of the form: jobfilename.o(jobid) jobfilename.e(jobid) Ex: simplejob.e555 – Contains STDERR simplejob.o555 – Contains STDOUT -j oe (combine standard output and standard error)

Ex2: Another sample batch script (pbs_script) #PBS -N hello #PBS -l mem=200mb #PBS -l walltime=0:15:00 #PBS -l nodes=2:ppn=2 #PBS -j oe #PBS –m abe mpirun –n 8 /nethome/users/ercal/MPI/hello This job “hello” requests 15 minutes of wall-time, 2 nodes using 2 processors each (4 processors), and 200MB of memory (100MB per node; 50MB per processor). Also, the output and error are written to one file. How many processes are created?

(This can help to check memory requirements before submitting a job) Tools qstat (jobid) qstat –u (userid) qstat -a This command returns the status of specified job(s). qdel (jobid) This command deletes a job. size executable_file_name gives O/P in the following format: text data bss dec hex filename 1650312 71928 6044136 7766376 768168 hello (This can help to check memory requirements before submitting a job)

Tips for programming in MPICH Use compiler optimizing flags for faster code. Some of them are: -O2 (moderate optimization) -funroll-loops (enables loop unrolling optimizations) -Wall (enables all common warnings) -ansi (enables ANSI C/C++ compliance) -pedantic (enables strictness of language compliance) Avoid using pointers in your program, unless absolutely necessary.

Tips (cont.) No scanf allowed in MPICH (fscanf is allowed). Instead, pass input to your program using command line arguments argc and argv in the mpiexec command of the PBS script Example: mpirun –n 8 /nethome/users/ercal/MPI/cpi …arguments… To give your job a high priority, set wall-time ≤ 15 minutes (#PBS -l walltime=0:15:00) and number of nodes ≤ 32 (#PBS -l nodes=16:ppn=2)