1. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM1 Introduction to the T3E Mark Durst NERSC/USG ERSUG Training, Argonne, IL 28 April 1999

2. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM2 Outline Hardware and Configuration Programming Environment Planning Runs Monitoring Execution Accounting Additional Resources Elvis Impression

3. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM3 NERSC T3E Configuration Commodity DEC Alpha EV-5 superscalar processor –450 MHz clock –900 Mflops/PE peak (only 5-10% typically achieved) Theoretical peak performance: 575 Gflops –256 MB memory per PE 692 PEs in 3 flavors –644 Application –33Command (ideally) –15OS Access via telnet, ssh, FTP Connect to NERSC mass storage, AFS

4. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM4 Interactive Environment UNICOS/mk Available shells: sh/ksh, csh, tcsh –csh: no file completion –tcsh not Cray-supported Home directories –2 GB file quota (with possible data migration) –3500 inode quota /usr/tmp –Used both for batch and temporary user space –75 GByte quota, 6K inode quota –Fastest transfer rates

5. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM5 modules modules manages user environment –Paths –Environment variables –Aliases Cray’s PrgEnv is modules -driven Provided startup files are critical! –Add to them, don’t clobber them –Add to paths, don’t set them –If you mess up, no compilers, etc. Largely automatic

6. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM6 More Fun with modules module list (tells you what’s loaded) module avail (lists them all) Other module subcommands –load –unload –switch –help Roll back compilers Test new versions http://home.nersc.gov/software/os/modules.html

7. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM7 Other modules imsl (loads by default) nag (loads by default) scalapack (1.5) GNU (prepends) and GNU.tools (appends) tools ( tcsh, bash ) netcdf KCC (KAI C++ compiler) USG tedi

8. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM8 Programming Environment f90 cc / CC cam (assembler) cld (loader; usually unneeded) pghpf KCC (“ module load KCC ”) totalview (debugger) pat, apprentice (performance analysis)

9. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM9 f90 Conforms to Fortran 90 standard Much “standard” f77 wasn’t User-defined and abstract types Array syntax Allocatable objects and pointers Additional intrinsics cpp -like preprocessor

10. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM10 Important f90 options -f : source form (fixed or free) Defaults:.f fixed,.f90 free -c : Compile only -o name: Name executable Overrides -c (use -b name instead) -g, -G0, -G1 : debugging -O[0-3] : general optimization -Ra, -Rb : Argument/Bounds checking -dp : Double precision  64-bit single precision -i 32 / -s default32 : 32-bit integers / numbers -ev : Static memory allocation

11. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM11 Executables: Malleable or Fixed -Xnpes (e.g., -X64 ) creates “fixed” executable –Always runs on same number of (application) processors –Type./a.out to run -Xm or no -X option creates “malleable” executable –./a.out will run on command PE – mpprun -n npes./a.out runs on npes APP PEs

12. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM12 Execution Model In F90, C, C++, all processors execute same program Can ask for: –Process number (from zero up) MY_PE() (F90) _my_pe() (C/C++) –Total number of PEs NUM_PES() (F90) _num_pes() (C/C++) Above used to establish “master/slave” relationships Libraries still needed for communication

13. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM13 Libraries MPI (Message-Passing Interface) PVM (Parallel Virtual Machine) SHMEM (SHared MEMory; non-portable) BLACS (Basic Linear Algebra Communication Subprograms) ScaLAPACK (SCAlable [parts of] LAPACK) NetCDF (NETwork Common Data Format) HDF (Hierarchical Data Format) LIBSCI (including parallel FFTs), NAG, IMSL

14. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM14 Archival Storage in HPSS High-Performance Storage System Designed for scalability & hierarchies User storage quotas exist Access via ftp or new hsi utility Two systems: –hpss.nersc.gov ( hsi hpss ) –archive.nersc.gov ( hsi, hsi archive ) contains old CFS files –merger planned

15. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM15 Networking Issues AFS –Accounts must be requested –Tiny local quotas –Available on Crays through NFS/AFS gateway Non-trivial latencies Remote logins –.rhosts access not permitted; no incoming “r- commands” –ssh available xterm only “backwards”

16. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM16 Execution modes Interactive serial –< 60 minutes –on command PEs –slightly reduced memory Interactive parallel –< 30 minutes –< 64 processors Batch

17. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM17 Batch queues on mcurie.nersc.gov To see them: qstat -b pe16 through pe512 –4 hours “on the torus” –Routine parallel jobs serial_short : 4 hours on a single command PE debug_small : ½ hour, up to 32 PEs long128, gc128, gc256 : 12-hour queues –  64 PEs – gc queues restricted Largest queues shuffled in at night Other jobs checkpointed out Subject to change

18. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM18 Example daily job mix

19. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM19 Batch submission Jobs are shell scripts cqsub submits, returns task ID; cqdel deletes cqstatl/qstat gets status (many options) NQS parameters determine queue –#QSUB -l mpp_p= … (number of PEs) –#QSUB -l mpp_t= … (“parallel” time) –for serial jobs: use #QSUB -q serial not #QSUB -l mpp_p=1

20. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM20 Pipe Queues You submit to pipe queues, not batch queues –Use only pipe names in directives like: #QSUB -q serial Group batch queues: – serial = serial_short – debug = debug_small – production = pe128 through pe512 – long = long128, gc128, gc256 3 jobs per user in production + long 3 in serial, one in debug To see them: qstat -p

21. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM21 Scheduling Information Lots of NQS-related limits –Queue run limits –Queue “complex” run limits Global Resource Manager –Fits jobs into contiguous sets of PEs –Once started, jobs run to completion (mostly) –First-fit algorithm lets small jobs trample big ones – grmview shows PE status, waiting jobs

22. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM22 Scheduling Information (cont’d) pslist gives summary of GRM data –No man page; pslist -h instead Checkpointing –For system maintenance –To run test and “grand challenge” jobs –Shows “ Hop ” in qstat/cqstatl (held by operator) mppview more nuts-and-bolts

23. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM23 Accounting and allocations T3E allocations are in node-minutes –setcub view repo=reponame –setcub view user=username newacct reponame switches repos interactively –One login name per user; multiple repos #QSUB -A reponame charges batch jobs Charging updated daily; enforcement manual

24. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM24 On-line Resources T3E pages under “Computers” at home.nersc.gov –Read overview once, check “Changes” monthly Docs in Cray on-line system – http://www.cray.com/swpubs/ “Topics” to T3E collection –Many other docs (e.g., F90, C manual sets) Cray Web site, www.cray.com –Technical documents, additional on-line docs NERSC T3E tutorials –“Training”  “NERSC Tutorials”

25. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM25 More on-line resources Other NERSC tutorials –Using the Cray f90 compiler at NERSC –Introduction to make –NQE: Using the batch system Look over NERSC Web generally

26. N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER INTRODUCTION TO THE T3E SYSTEM26 man pages cqsub cqstatl f90 cc CC