1. Unified Parallel C at NERSC
Kathy Yelick
EECS, U.C. Berkeley and NERSC/LBNL
UPC Team: Dan Bonachea, Jason Duell, Paul Hargrove, Parry Husbands, Costin Iancu, Mike Welcome, Christian Bell

2. Motivation for a new class of languages
Outline
Motivation for a new class of languages
Programming models
Architectural trends
Overview of Unified Parallel C (UPC)
Programmability advantage
Performance opportunity
Status
Next step
Related projects

3. Programming Model 1: Shared Memory
Program is a collection of threads of control.
Many languages allow threads to be created dynamically,
Each thread has a set of private variables, e.g. local variables on the stack.
Collectively with a set of shared variables, e.g., static variables, shared common blocks, global heap.
Threads communicate implicitly by writing/reading shared variables.
Threads coordinate using synchronization operations on shared variables
Like concurrent programming on a uniprocessor.
x = ...
Shared
y = ..x ... P1
Private
. . . P0 Pn

4. Programming Model 2: Message Passing
Program consists of a collection of named processes.
Usually fixed at program startup time
Thread of control plus local address space -- NO shared data.
Logically shared data is partitioned over local processes.
Processes communicate by explicit send/receive pairs
Coordination is implicit in every communication event.
MPI is the most common example
send P0,X
recv Pn,Y Y P1 X
. . . P0 Pn

5. Advantages/Disadvantages of Each Model
Shared memory
Programming is easier
Can build large shared data structures
Machines don’t scale
SMPs typically < 16 processors (Sun, DEC, Intel, IBM)
Distributed shared memory < 128 (SGI)
Performance is hard to predict and control
Message passing
Machines easier to build from commodity parts
Can scale (given sufficient network)
Programming is harder
Distributed data structures only in the programmers mind
Tedious packing/unpacking of irregular data structures

6. Global Address Space Programming
Intermediate point between message passing and shared memory
Program consists of a collection of processes.
Fixed at program startup time, like MPI
Local and shared data, as in shared memory model
But, shared data is partitioned over local processes
Remote data stays remote on distributed memory machines
Processes communicate by reads/writes to shared variables
Examples are UPC, Titanium, CAF, Split-C
Note: These are not data-parallel languages
heroic compilers not required

7. GAS Languages on Clusters of SMPs
SMPs are the fastest commodity machine, so used as a node in large-scale clusters
Common names:
CLUMP = Cluster of SMPs
Hierarchical machines, constellations
Most modern machines look like this:
Millennium, IBM SPs, (not the t3e)...
What is an appropriate programming model?
Use message passing throughout
Unnecessary packing/unpacking overhead
Hybrid models
Write 2 parallel programs (MPI + OpenMP or Threads)
Global address space
Only adds test (on/off node) before local read/write

8. Top 500 Supercomputers
Listing of the 500 most powerful computers in the world
- Yardstick: Rmax from LINPACK MPP benchmark
Ax=b, dense problem
- Dense LU Factorization (dominated by matrix multiply)
Updated twice a year SC‘xy in the States in November
Meeting in Mannheim, Germany in June
All data (and slides) available from
Also measures N-1/2 (size required to get ½ speed)
performance
Rate
Size

11. Motivation for a new class of languages
Outline
Motivation for a new class of languages
Programming models
Architectural trends
Overview of Unified Parallel C (UPC)
Programmability advantage
Performance opportunity
Status
Next step
Related projects

12. Parallelism Model in UPC
UPC uses an SPMD model of parallelism
A set if THREADS threads working independently
Two compilation models
THREADS may be fixed at compile time or
Dynamically set at program startup time
MYTHREAD specifies thread index (0..THREADS-1)
Basic synchronization mechanisms
Barriers (normal and split-phase), locks
What UPC does not do automatically:
Determine data layout
Load balance – move computations
Caching – move data
These are intentionally left to the programmer

13. Shared and Private Variables in UPC
A shared variable has one instance, shared by all threads.
Affinity to thread 0 by default (allocated in processor 0’s memory)
A private variable has an instance per thread
Example:
int x; // private copy for each processor
shared int y; // one copy on P0, shared by all others
x = 0; y = 0;
x += 1; y += 1;
After executing this code
x will be 1 in all threads; y will be between 1 and THREADS
Shared scalar variable are somewhat rare because:
cannot be automatic (declared in a function) (Why not?)

14. UPC Pointers Pointers may point to shared or private variables
Same syntax for use, just add qualifier
shared int *sp;
int *lp;
sp is a pointer to an integer residing in the shared memory space.
sp is called a shared pointer (somewhat sloppy).
x: 3
Shared
sp:
sp:
sp:
Global address space
Private
lp:

15. Shared Arrays in UPV
Shared array elements are spread across the threads
shared int x[THREADS] /*One element per thread */
shared int y[3][THREADS] /* 3 elements per thread */
shared int z[3*THREADS] /* 3 elements per thread, cyclic */
In the pictures below
Assume THREADS = 4
Elements with affinity to processor 0 are red
Of course, this is really a 2D array x
y blocked
z cyclic

16. Work Sharing with upc_forall()
Iterations are independent
Each thread gets a bunch of iterations
Simple C-like syntax and semantics
upc_forall(init; test; loop; affinity)
statement;
Affinity field to distribute the work
Round robin
Chunks of iterations
Semantics are undefined if there are dependencies between iterations
Programmer has indicated iterations are independent

17. Vector Addition with upc_forall
The loop in vadd is common, so there is upc_forall:
4th argument is int expression that gives “affinity”
Iteration executes when:
affinity%THREADS is MYTHREAD
/* vadd.c */
#include <upc_relaxed.h> #define N 100*THREADS shared int v1[N], v2[N], sum[N]; void main() { int i; upc_forall(i=0; i<N; i++; i)
sum[i]=v1[i]+v2[i]; }

18. Layouts in General
All non-array objects have affinity with thread zero.
Array layouts are controlled by layout specifiers.
layout_specifier::
null
layout_specifier [ integer_expression ]
The affinity of an array element is defined in terms of the
block size, a compile-time constant, and THREADS a runtime constant.
Element i has affinity with thread
( i / block_size) % PROCS.

19. 2D Array Layouts in UPC
Array a1 has a row layout and array a2 has a block row layout.
shared [m] int a1 [n][m];
shared [k*m] int a2 [n][m];
If (k + m) % THREADS = = 0 them a3 has a row layout
shared int a3 [n][m+k];
To get more general HPF and ScaLAPACK style 2D blocked layouts, one needs to add dimensions.
Assume r*c = THREADS;
shared [b1][b2] int a5 [m][n][r][c][b1][b2];
or equivalently
shared [b1*b2] int a5 [m][n][r][c][b1][b2];

20. Domain Decomposition for UPC
Exploits locality in matrix multiplication
A (N  P) is decomposed row-wise into blocks of size (N  P) / THREADS as shown below:
B(P  M) is decomposed column wise into M/ THREADS blocks as shown below:
Thread THREADS-1
Thread 0 P M
0 .. (N*P / THREADS) -1
Thread 0
(N*P / THREADS)..(2*N*P / THREADS)-1
Thread 1 N P
((THREADS-1)N*P) / THREADS .. (THREADS*N*P / THREADS)-1
Thread THREADS-1
Note: N and M are assumed to be multiples of THREADS
Columns 0: (M/THREADS)-1
Columns ((THREAD-1)  M)/THREADS:(M-1)

21. UPC Matrix Multiplication Code
/* mat_mult_1.c */
#include <upc_relaxed.h>
shared [N*P /THREADS] int a[N][P], c[N][M];
// a and c are row-wise blocked shared matrices
shared[M/THREADS] int b[P][M]; //column-wise blocking
void main (void) {
int i, j , l; // private variables
upc_forall(i = 0 ; i<N ; i++; &c[i][0]) {
for (j=0 ; j<M ;j++) {
c[i][j] = 0;
for (l= 0 ; lP ; l++) c[i][j] += a[i][l]*b[l][j]; }

22. Notes on the Matrix Multiplication Example
The UPC code for the matrix multiplication is almost the same size as the sequential code
Shared variable declarations include the keyword shared
Making a private copy of matrix B in each thread might result in better performance since many remote memory operations can be avoided
Can be done with the help of upc_memget

23. Overlapping Communication in UPC
Programs with fine-grained communication require overlap for performance
UPC compiler does this automatically for “relaxed” accesses.
Acesses may be designated as strict, relaxed, or unqualified (the default).
There are several ways of designating the ordering type.
A type qualifier, strict or relaxed can be used to affect all variables of that type.
Labels strict or relaxed can be used to control the accesses within a statement.
strict : { x = y ; z = y+1; }
A strict or relaxed cast can be used to override the current label or type qualifier.

24. Performance of UPC Reason why UPC may be slower than MPI
Shared array indexing is expensive
Small messages encouraged by model
Reasons why UPC may be faster than MPI
MPI encourages synchrony
Buffering required for many MPI calls
Remote read/write of a single word may require very little overhead
Cray t3e, Quadrics interconnect (next version)
Assuming overlapped communication, the real issues is overhead: how much time does it take to issue a remote read/write?

25. UPC versus MPI for Edge detection
a. Execution time
b. Scalability
Performance from Cray T3E
Benchmark developed by El Ghazawi’s group at GWU

26. UPC versus MPI for Matrix Multiplication
a. Execution time
b. Scalability
Performance from Cray T3E
Benchmark developed by El Ghazawi’s group at GWU

27. UPC vs. MPI for Sparse Matrix-Vector Multiply
Short term goal:
Evaluate language and compilers using small applications
Longer term, identify large application
Show advantage of t3e network model and UPC
Performance on Compaq machine worse:
Serial code
Communication performance
New compiler just released

28. Particle/Grid Methods in UPC ?
Experience so far in a related language
Titanium, Java-based GAS language
Immersed boundary method
Most time in communication between mesh and particles
Currently uses bulk communication
May benefit from SPMV trick

29. EM3D Performance in Split-C Language on CM-5
Maxwells Equations on an Unstructured 3D Mesh: Explicit Method
Irregular Bipartite Graph of varying degree
(about 20) with weighted edges v1 v2 w1 w2 H E B
Basic operation is to subtract weighted sum of
neighboring values
for all E nodes
for all H nodes D

30. Split-C: Performance Tuning on the CM5
Tuning affects application performance

31. Motivation for a new class of languages
Outline
Motivation for a new class of languages
Programming models
Architectural trends
Overview of Unified Parallel C (UPC)
Programmability advantage
Performance opportunity
Status
Next step
Related projects

32. UPC Implementation Effort
UPC efforts elsewhere
IDA: t3e implementation based on old gcc
GMU (documentation) and UMC (benchmarking)
Compaq (Alpha cluster and C+MPI compiler (with MTU))
Cray, Sun, and HP (implementations)
Intrepid (SGI compiler and t3e compiler)
UPC Book:
T. El-Ghazawi, B. Carlson, T. Sterling, K. Yelick
Three components of NERSC effort
Compilers (SP and PC clusters) + optimization (DOE)
Runtime systems for multiple compilers (DOE + NSA)
Applications and benchmarks (DOE)

33. NERSC compiler (Costin Iancu)
Compiler Status
NERSC compiler (Costin Iancu)
Based on Open64 compiler for C
Parses and type-checks UPC
Code generation for SMPs underway
Generate C on most machines, possibly IA64 later
Investigating optimization opportunities
Focus of this compiler is high level optimizations
Intrepid compiler
Based on gcc (3.x)
Will target our runtime layer on most machines
Initial focus is t3e, then Pentium clusters

34. Runtime System Characterizing network performance
Low latency (low overhead) -> programmability
Optimization depend on network characteristics
T3e was ideal
Quadrics reports very low overhead coming
Difficult to access low level SP and Myrinet

35. Undertake larger application effort What type of application?
Next Step
Undertake larger application effort
What type of application?
Challenging to write in MPI (e.g., sparse direct solvers)
Irregular communication (e.g., PIC)
Well-understood algorithm

36. Motivation for a new class of languages
Outline
Motivation for a new class of languages
Programming models
Architectural trends
Overview of Unified Parallel C (UPC)
Programmability advantage
Performance opportunity
Status
Next step
Related projects

37. 3 Related Projects on Campus
Titanium
High performance Java dialect
Collaboration with Phil Colella and Charlie Peskin
BeBOP: Berkeley Benchmarking and Optimization
Self-tuning numerical kernels
Sparse matrix operations
Pyramid mesh generator (Jonathan Shewchuk)

38. Locality and Parallelism
Proc
Cache
L2 Cache
L3 Cache
Memory
Conventional
Storage
Hierarchy
potential
interconnects
Large memories are slow, fast memories are small.
Storage hierarchies are large and fast on average.
Parallel processors, collectively, have large, fast memories -- the slow accesses to “remote” data we call “communication”.
Algorithm should do most work on local data.

39. Tuning pays off – ATLAS (Dongarra, Whaley)
Extends applicability of PHIPAC
Incorporated in Matlab (with rest of LAPACK)

40. Speedups on SPMV from Sparsity on Sun Ultra 1/170 – 1 RHS

41. Speedups on SPMV from Sparsity on Sun Ultra 1/170 – 9 RHS

42. Exploit Itanium Architecture
Future Work
Exploit Itanium Architecture
128 (82-bit) floating point registers
9 HW formats: 24/8(v), 24/15, 24/17, 53/11, 53/15, 53/17, 64/15, 64/17
Many few load/store instructions
fused multiply-add instruction
predicated instructions
rotating registers for software pipelining
prefetch instructions
three levels of cache
Tune current and wider set of kernels
Improve heuristics, eg choice of r x c
Incorporate into
SUGAR
Information Retrieval
Further automate performance tuning
Generation of algorithm space generators