1. University of Illinois at Urbana-Champaign Using the NCSA Supercluster for Cactus NT Cluster Group Computing and Communications Division NCSA Mike Showerman mshow@ncsa.uiuc.edu

2. University of Illinois at Urbana-Champaign NCSA NT 320 Pentium® CPU Cluster 64 CPUs - Serial 32 Compaq PWS 6000 Dual 333 MHz Pentium II 512 MB memory 256 CPUs - Parallel MPI 64 HP Kayak XU systems Dual 550 MHz Pentium III Xeon 1 GB RAM 64 HP Kayak XU systems Dual 300MHz Pentium II 512 MB memory 64 Dual 550 MHz Pentium III Xeon HP Kayaks back-to-back

3. University of Illinois at Urbana-Champaign NT Cluster Software Microsoft NT 4.0 Server LSF from Platform for queuing system Cygnus toolkit –UNIX tools for porting MPI –HPVM from Chien’s CSAG group Microsoft and Digital compilers –C, C++, F77, F90 OpenMP –This does not span hosts. 2 processor only

4. University of Illinois at Urbana-Champaign Cactus Performance on 32 Processors Source: http://origin.aei-potsdam.mpg.de/projects/cluster by O. Wehrens, NCSA NT group for NT 550 results

5. University of Illinois at Urbana-Champaign Obtaining an Account Cluster is in “friendly user” mode –No production level support –System upgrades are common –Your research must be tolerant of this environment –Allocation data collected, but not currently billed Contact NCSA consulting to get an account –Details at http://www.ncsa.uiuc.edu/SCD/Hardware/NTCluster/

6. University of Illinois at Urbana-Champaign Accessing the Cluster Windows Terminal Server Interactive Nodes –Multiuser form of Windows NT –Surprisingly good performance –Access Methods –Windows RDP Client from Microsoft –Available at http://ntsc-ts1.ncsa.uiuc.edu –Citrix ICA client –Available for most platforms –http://www.citrix.com to download the clients –A java applet client is available at http://ntsc-ts1.ncsa.uiuc.eduhttp://ntsc-ts1.ncsa.uiuc.edu –X Windows –Rsh ntsc-ts1.ncsa.uiuc.edu wincenter –Option: -resolution ex: -resolution 800x600 –Option: -depth for color depth ex: -depth 8

7. University of Illinois at Urbana-Champaign Windows NT on a Web Page

8. University of Illinois at Urbana-Champaign Compiling Your Executable Choose the desired Terminal server client –Connect to ntsc-ts1.ncsa.uiuc.edu Open a command prompt –Select “run” from the START menu. –Enter cmd in the box Hit OK Start bash –Type “cygnus” at the prompt –This is the Cygnus Windows toolkit –Tcsh is available for csh scripts Run make –Equivalent to gmake Unix style commands –tar, make, awk, grep, ar, ranlib, cp, rm etc.

9. University of Illinois at Urbana-Champaign Modifying your Makefile Editors available –Windows style – notepad and write –Unix style – vi and emacs Compilers –C/C++ = cl –FORTRAN = df -> f77 and f90 Linker –LINK or use the compiler Options available Cl /? Df /? or link /? to list options

10. University of Illinois at Urbana-Champaign Building an Application Makefile differences –Set compiler to be cl or df –Change extension on object files (.obj by default) –For MPI set libs –mpi.lib fm.lib wsock32.lib kernel32.lib advapi32.lib –LIB and include paths are already set in environment –May also need dfport for FORTRAN applications Suffix rules for makefiles –.c.o : ; $(CC) -c $(CCFLAGS) $(CDEFS) $< -Fo$*.o –.f.o : ; $(F77) -c $(F77FLAGS) $< -Fo$*.o Link –Link objects.o /OUT:example.exe

11. University of Illinois at Urbana-Champaign bsub Command Bsub –Current working directory is passed through LSF –Your current working directory will be the UNC path of your network drive ex: bsub from z:\bob will have a cwd of \\ntsc-file1\home\bob –Arguments –-q qname –Current queues are 48NoMPI - no myrinet, single processor or machine applications 128hp - 300 MHz HP systems for parallel work, up to 128p 128hp550 - 550 MHz HP systems for >32p applications, up to 128p –-n is for LSF to allocate N machines –-np is for MPI to use N machines –-o output file (for standard out)

12. University of Illinois at Urbana-Champaign bsub Command (cont.) Use prun to start the processes –\\ntsc-file1\home\lsf\prun runs the trailing command on all\\ntsc-file1\home\lsf\prun of the hosts allocated Example Command –bsub -q 128hp -o out.txt -n 32 \lsf\prun helloworld -np 32 -key UniqueText If you use standard input –Wrap your command with cmd /c “type file|cmd“ –Example –Z:\Bsub –n 32 –o out –q 128hp \lsf\prun cmd /c “type input.txt | helloworld –np 32 –key test32” if using bash, escape all \ with another –Bash$ bsub -q 128hp -o out -n N \\lsf\\prun exec args -np N -key ABC

13. University of Illinois at Urbana-Champaign Interacting with jobs Your jobs can be listed by bjobs –Bjobs –u all shows all jobs by all users –Bjobs –l provides detailed job information Bkill –Sends a control-c to your processes Bpeek –Outputs the current job output (stdout) Bqueues –Information about the queues (pending jobs…) Lsload –Provides current information about the hosts LSF DOCS are on http://ntsc-file1.ncsa.uiuc.edu

14. University of Illinois at Urbana-Champaign Available at: \\ntsc-file1\glmon\glmon (this can be entered from the command line or “run” Box) Shows in real time: machine status systems status current load load by user name Load by jobid All running/pending jobs NT Cluster Monitoring

15. University of Illinois at Urbana-Champaign Advanced Topics Changing passwords –Log on and select “Windows security” from the start menu. –Select “change password” –Also, You must change your password for LSF Run lspasswd from a command prompt You must type in the same password as your NT password Storage –Currently there are no quotas in place –Your Windows Desktop –Please do not store data here –Your home directory –Located in \\ntsc-file1\home and mapped to z:\ on the terminal server –This area is backed up and is used to store semi-permanent data –Scratch –Located at \\ntsc-file2\scratch Larger data sets can be stored here\\ntsc-file2\scratch

16. University of Illinois at Urbana-Champaign Advanced Topics (cont.) Using scratch –Map a drive letter for scratch –Net use y: \\ntsc-file2\scratch\\ntsc-file2\scratch net use y: \\\\ntsc-file2\\scratch under bash\\\\ntsc-file2\\scratch –Move to the y: drive by typing y: at a command prompt Cd //y using bash –Create a directory and move data here –Submit the job from this drive Getting your files –The terminal servers have both ftp and Kerberos ftp clients –See http://www.ncsa.uiuc.edu/SCD/Hardware/NTCluster/

17. University of Illinois at Urbana-Champaign Cactus on the cluster Getting the source –From a command prompt type the following –Cvs –d :pserver:cvs_anon@cvs.cactuscode.org:/cactus checkout Source Modification –Due to our MPI (fix is on the way) –In Cactus/src/main/ProcessCommandLine.c –Line 85 remove CCTKi_CommandLineHelp(); Open a bash shell in the cactus directory –Make checkout to get thorns –Make mpi MPI=CUSTOM MPI_LIBS=“mpi.lib fm.lib advapi32.lib kernel32.lib wsock32.lib” MPI_INC_DIRS=“d:/apps/hpvm/include” –Make mpi –done

18. University of Illinois at Urbana-Champaign Future Directions –Better integration with the mass storage system –High performance shared filesystems –Improved reliability and process management –Advanced user support –Advancements in interconnects –Better scaling –Better performance