UPC at NERSC/LBNL Kathy Yelick, Christian Bell, Dan Bonachea, Yannick Cote, Jason Duell, Paul Hargrove, Parry Husbands, Costin Iancu, Mike Welcome NERSC, U.C. Berkeley, and Concordia U.

Overview of NERSC Effort Three components: Compilers IBM SP platform and PC clusters are main targets Portable compiler infrasturucture (UPC->C) Optimization of communication and global pointers Runtime systems for multiple compilers Allow use by other languages (Titanium and CAF) And in other UPC compilers Performance evaluation Applications and benchmarks Currently looking at NAS PB Evaluating language and compilers Plan to do a larger application next year

NERSC UPC Compiler Compiler being developed by Costin Iancu Status Based on Open64 compiler for C Originally developed at SGI Has IA64 backend with some ongoing development Software available on SourceForge Can use as C to C translator Can either generate before most optimizations Or after, but this is known to be buggy right now Status Parses and type-checks UPC Finishing code generation for UPC->C translator Code generation for SMPs underway

Compiler Optimizations Based on lessons learned from Titanium: UPC in Java Split-C: one of the UPC predecessors Optimizations Pointer optimizations: Optimization of phase-less pointers Turn global pointers into local ones Overlap Split-phase Merge “synchs” at barrier Aggregation Split-C data on CM-5

Portable Runtime Support Developing a runtime layer that can be easily ported and tuned to multiple architectures. Direct implementations of parts of full GASNet UPCNet: Global pointers (opaque type with rich set of pointer operations), memory management, job startup, etc. Generic support for UPC, CAF, Titanium GASNet Extended API: Supports put, get, locks, barrier, bulk, scatter/gather GASNet Core API: Small interface based on “Active Messages” Core sufficient for functional implementation

Portable Runtime Support Full runtime designed to be used by multiple compilers NERSC compiler based on Open64 Intrepid compiler based on gcc Communication layer designed to run on multiple machines Hardware shared memory (direct load/store) IBM SP (LAPI) Myrinet 2K (GM) Quadrics (Elan3) Dolphin VIA and Infiniband in anticipation of future networks MPI for portability Use communication micro-benchmarks to choose optimizations

Possible Optimizations Use of lightweight communication Converting reads to writes (or reverse) Overlapping communication with communication Overlapping communication with computation Aggregating small messages into larger ones

MPI vs. LAPI on the IBM SP LAPI generally faster than MPI Non-Blocking (relaxed) faster than blocking

Overlapping Computation: IBM SP Nearly all software overhead – no computation overlap Recall: 36 usec blocking, 12 usec nonblocking

Conclusions for IBM SP LAPI is better the MPI Reads/Writes roughly the same cost Overlapping communication with communication (pipelining) is important Overlapping communication with computation Important if no communication overlap Minimal value if >= 2 messages overlapped Large messages are still much more efficient Generally noisy data: hard to control

Other Machines Observations: Low latency reveals programming advantage T3E is still much better than the other networks usec

Applications Status Short term goal: Evaluate language and compilers using small applications Longer term, identify large application Conjugate Gradient Show advantage of t3e network model and UPC Performance on Compaq machine worse: Serial code Communication performance Simple n2 particle simulation Currently working on NAS MG Need for shared array arithmetic optimizations

Future Plans This month This year Next year Draft of runtime spec Draft of GASNet spec This year Initial runtime implementation on shared memory Runtime implementation on distributed memory (M2K, SP) NERSC compiler release 1.0b for IBM SP Next year Compiler release for PC cluster Development of CLUMP compiler Begin large application effort More GASNet implementations Advanced analysis and optimizations

Runtime Breakout How many runtime systems? Language issues Compaq MTU LBNL/Intrepid Language issues Locks Richard Stallman’s ? upc_phaseof for pointers with indef. block size Misc Runtime extensions Strided and scatter/gather memcopy

Read/Write Behavior Negligible difference between blocking read and write performance

Overlapping Communication Effects of pipelined communication are significant 8 overlapped messages are sufficient to saturate NI Queue depth

Overlapping Computation Same experiment, but fix total amount of computation

SPMV on Compaq/Quadrics Seeing 15 usec latency for small msgs Data for 1 thread per node Note: this data was hand-typed a picture from Parry – couldn’t cut and paste

Optimization Strategy Optimizations of communication is key to making UPC more usable Two problems: Analysis of code to determine which optimizations are legal Use of performance models to select transformations to improve performance Focus on the second problem here

Runtime Status Characterizing network performance Low latency (low overhead) -> programmability Specification of portable runtime Communication layer (UPC, Titanium, Co-Array Fortran) Built on small “core” layer; interoperability a major concern Full runtime has memory management, job startup, etc. usec

What is UPC? UPC is an explicitly parallel language Global address space; can read/write remote memory Programmer control over layout and scheduling From Split-C, AC, PCP Why a new language? Easier to use than MPI, especially for program with complicated data structures Possibly faster on some machines, but current goal is comparable performance p0 p1 p2

Background UPC efforts elsewhere UPC Book: IDA: t3e implementation based on old gcc GMU (documentation) and UMC (benchmarking) Compaq (Alpha cluster and C+MPI compiler (with MTU)) Cray, Sun, and HP (implementations) Intrepid (SGI compiler and t3e compiler) UPC Book: T. El-Ghazawi, B. Carlson, T. Sterling, K. Yelick Three components of NERSC effort Compilers (SP and PC clusters) + optimization (DOE/UPC) Runtime systems for multiple compilers (DOE/Pmodels + NSA) Applications and benchmarks (DOE/UPC)

Overlapping Computation on Quadrics 8-Byte non-blocking put on Compaq/Quadrics