1. Parallel Programming using the PGAS Approach

2. UPC (Unified Parallel C)
Outline
Introduction
Programming parallel systems: threading, message passing
PGAS as a middle ground
UPC (Unified Parallel C)
History of UPC
Shared scalars and arrays
Work-sharing in UPC: parallel loop
DASH – PGAS in the form of a C++ template library
A quick overview of the project
Conclusion

3. Programming Parallel Machines
The two most widely used approaches for parallel programming:
Shared Memory Programming
using Threads
Message Passing
Memory System … …
Mem
Process/thread
Physical memory
Memory Access (read/write)
Explicit Message
Private data
Shared data

4. Shared Memory Programming using Threads
Examples:
OpenMP, Pthreads, C++ threads, Java threads
Limited to shared memory systems
Shared data can be directly accessed
Implicit communication, direct reads, writes
Advantages
Typically easier to program, natural extension of sequential programming
Disadvantages
Subtle bugs, race conditions
False sharing as a performance problem
Memory System …
Process/thread
Physical memory
Memory Access (read/write)
Explicit Message
Private data
Shared data

5. Message Passing Example Disadvantages Advantages
MPI (message passing interface)
Disadvantages
Complex programming paradigm
Manual data partitioning required
Explicit coordination of communication (send/receive pairs)
Data replication (memory requirement)
Advantages
Highly efficient and scalable (to the largest machines in use today)
Data locality is “automatic”
No false sharing, no race conditions
Runs everywhere …
Mem
Process/thread
Physical memory
Memory Access (read/write)
Explicit Message
Private data
Shared data

6. Partitioned Global Address Space
Best of both worlds
Can be used on large scale distributed memory machines but also on shared memory machines
A PGAS program looks much like a regular threaded program, but
Sharing data is declared explicitly
The data partitioning is made explicit
Both needed for performance!
PGAS Layer
shared data space
is partitioned!
Process/thread
Physical memory
Memory Access (put/get)
Explicit Message
Private data
Shared data

7. Partitioned Global Address Space
Example
Let’s call the members of our program threads
Let’s assume we use the SPMD (single program multiple data) paradigm
Let’s assume we have a new keyword “shared” that puts variables in the shared global address space
This is how PGAS is ex-
pressed in UPC
(Unified Parallel C)! (more later)
shared int ours;
int mine;
Global Address Space
Private
Shared
Thread 0
Thread 1
Thread n-1 …
mine
ours
n copies of mine (one per thread)
Each thread can only access its own copy
1 copy of ours
Accessible by every thread

8. Example: a shared array (UPC)
Shared Arrays
Example: a shared array (UPC)
shared int[4] ours;
int mine;
Thread 0
Thread 1
Thread 3
mine
ours[0]
Thread 2
ours[1]
ours[2]
ours[3]
Shared
Global Address Space
Private
Affinity – in which partition a data item “lives”
ours (previous slide) lives in partition 0 (by convention)
ours[i] lives in partition i

9. Local-view vs. Global-view
Two ways to organize access to shared data:
Global-view E.g., Unified Parallel C
Local-view E.g., Co-array Fortran
X is declared in terms of its global size
X is accessed in terms of global indices
process is not specified explicitly
shared int X[100];
X[i]=23;
Global size,
Global index
a, b are declared in terms of their local size
a,b are accessed in terms of local indices
process (image) is specified explicitly (the co-index)
integer :: a(100)[*], b(100)[*]
b(17) = a(17)[2]
Local size,
Local index
co-dimension / co-index
in square brackets

10. UPC is an extension to ANSI C
UPC History and Status
UPC is an extension to ANSI C
New keywords, library functions
Developed in the late 1990s and early 2000s
Based on previous projects at UCB, IDA, LLNL, …
Status
Berkeley UPC
GCC version
Vendor compilers (Cray, IBM, …)
Most often used on graph problems, irregular parallelism

11. A number of threads working independently in a SPMD fashion
UPC Execution Model
A number of threads working independently in a SPMD fashion
Number of threads specified at compile-time or run-time; available as program variable THREADS
Note: “thread” is the UPC terminology. UPC threads are most often implemented as a full OS processes
MYTHREAD specifies thread index (0...THREADS-1)
upc_barrier is a global synchronization: all wait before continuing
There is a form of parallel loop (later)
There are two compilation modes
Static threads mode:
THREADS is specified at compile time by the user
The program may use THREADS as a compile-time constant
Dynamic threads mode:
Compiled code may be run with varying numbers of thread

12. Any legal C program is also a legal UPC program SPMD Model
Hello World in UPC
Any legal C program is also a legal UPC program
SPMD Model
If you compile and run it as UPC with N threads, it will run N copies of the program (same model as MPI)
#include <upc.h> /* needed for UPC extensions */
#include <stdio.h>
main() {
printf("Thread %d of %d: hello UPC world\n",
MYTHREAD, THREADS); }
Thread 0 of 4: hello UPC world
Thread 1 of 4: hello UPC world
Thread 3 of 4: hello UPC world
Thread 2 of 4: hello UPC world

13. A Bigger Example in UPC: Estimate p
Estimate Pi by throwing darts at a unit square
Calculate percentage that fall in the unit circle
Area of square = r2 = 1
Area of circle quadrant = ¼ p r2 = p/4
Randomly throw darts at (x,y) positions
If x2 + y2 < 1, then point is inside circle
Compute ratio R:
R = # points inside / # points total
p ≈ 4 R
r =1

14. Pi in UPC, First version #include <stdio.h>
#include <math.h>
#include <upc.h>
main(int argc, char *argv[]) {
int i, hits, trials = 0;
double pi;
if (argc != 2) trials = ;
else trials = atoi(argv[1]);
srand(MYTHREAD*17);
for (i=0; i < trials; i++) hits += hit();
pi = 4.0*hits/trials;
printf("PI estimated to %f.", pi); }
Each thread gets its own copy of these variables
Each thread can use the input arguments
Initialize RNG in math library
hit() : get random numbers and return 1 if inside circle
This program computes N independent estimates of Pi (when run with N threads)

15. Pi in UPC, Shared Memory Style
shared variable to
record hits
shared int hits=0;
main(int argc, char **argv) {
int i, my_trials = 0;
int trials = atoi(argv[1]);
my_trials = (trials +
THREADS-1)/THREADS;
srand(MYTHREAD*17);
for (i=0; i < my_trials; i++)
hits += hit();
upc_barrier();
if (MYTHREAD == 0) {
printf("PI estimated to %f.", 4.0*hits/trials); }
divide up work evenly
accumulate hits
There is a problem with this program…
Problem with this program: race condition! Reading/writing to hits is not synchronized

16. Fixing the Race Condition
A possible fix for the race condition
Have a separate counter per thread (use a shared array)
One thread computes the total sum
int hits=0;
shared int all_hits[THREADS];
main(int argc, char **argv) {
// declarations and initialization code omitted
for (i=0; i < my_trials; i++)
all_hits[MYTHREAD] += hit();
upc_barrier();
if (MYTHREAD == 0) {
for (i=0; i < THREADS; i++) hits += all_hits[i];
printf("PI estimated to %f.", 4.0*hits/trials); }
Shared array: 1 element per thread
Each thread accesses its
local element, no race condition, no remote communication
Thread 0 computes overall sum

17. Customizable layout of one and multi-dimensional arrays
Other UPC Features
Locks
upc_lock_t: pairwise synchronization between threads
Can also be used to fix race condition in previous example
Customizable layout of one and multi-dimensional arrays
Blocked, cyclic, block-cyclic; cyclic is the default
Split-phase barrier
upc_notify() and upc_wait() instead of upc_barrier()
Shared pointer and pointer to shared
Work-sharing (parallel loop)

18. Worksharing: Vector Addition Example
#include <upc_relaxed.h>
#define N 100*THREADS
shared int v1[N], v2[N], sum[N];
void main() {
int i;
for(i=0; i<N; i++) {
if(MYTHREAD == i%THREADS) {
sum[i]=v1[i]+v2[i]; }
Default layout: cyclic (round robin) 1 2 3 v1 … 1 2 3 v2 … 1 2 3
sum …
Access local elements only:
sum[i] has affinity to thread i
Each thread iterates over the indices that it “owns”
This is a common idiom called “owner computes”
UPC supports this idiom directly with a parallel version of the for loop: upc_forall

19. UPC work-sharing with forall
upc_forall(init; test; loop; affinity)
statement;
upc_forall
init, test, loop: same as regular C for loop: defines loop start, increment, and end
affinity: defines which iterations a thread is responsible for
Syntactic sugar for loop on previous slide:
Loop over all
Work on those with affinity to this thread
Programmer guarantees that the iterations are independent
Undefined if there are dependencies across threads
Affinity expression: two options
Integer: affinity%THREADS is MYTHREAD
Pointer: upc_threadof(affinity) is MYTHREAD

20. Vector Addition with upc_forall
The vector addition example can be rewritten as follows
Equivalent code could use „&sum[i]“ for the affinity test
The code would be correct (but slow) if the affinity expression is i+1 rather than i.
#define N 100*THREADS
shared int v1[N], v2[N], sum[N];
void main() {
int i;
upc_forall(i=0; i<N; i++; i)
sum[i]=v1[i]+v2[i]; }
Affinity expression

21. UPC is an extention to C, implementing the PGAS model
UPC Summary
UPC is an extention to C, implementing the PGAS model
Available as a gcc version, Berkeley UPC, from some vendors
Today most often used for graph problems, irregular parallelism
PGAS is a concept realized in UPC and other languages
Co-array Fortran, Titanium
Chapel, X10, Fortress (HPCS languages)
Not covered
Collective operations (reductions, etc. similar to MPI)
Dynamic memory allocation in shared space
UPC shared pointers

22. DASH – PGAS in the form of a C++ Template library
DASH – Overview
DASH – PGAS in the form of a C++ Template library
Focus on data structures
Array a can be stored in the memory of several nodes
a[i] transparently refers to local memory or to remote memory via operator overloading
dash::array<int> a(1000);
a[23]=412;
std::cout<<a[42]<<std::endl;
Not a new language to learn
Can be integrated with existing (MPI) applications
Support for hierarchical locality
Team hierarchies and locality iterators
Node
e.g., STL vector,
array
DASH array

23. Hierarchical Machines
Machines are getting increasingly hierarchical
Both within nodes and between nodes
Data locality is the most crucial factor for performance and energy efficiency
Source: LRZ SuperMUC system description.
Source: Bhatele et al.: Avoiding hot-spots in two-level direct networks. SC 2011.
Source: Steve Keckler et al.: Echelon System Sketch
Hierarchical locality not well supported by current approaches. PGAS languages usually only offer a two-level differentiation (local vs. remote).

24. DASH – Overview and Project Partners
LMU Munich (K. Fürlinger)
HLRS Stuttgart (J. Gracia)
TU Dresden (A. Knüpfer)
KIT Karlsruhe (J. Tao)
CEODE Beijing (L. Wang, associated)
DASH Runtime
DASH C++ Template Library
DASH Application
Tools and Interfaces
Hardware: Network, Processor,
Memory, Storage
One-sided Communication
Substrate
MPI
GASnet
GASPI
ARMCI
Component of DASH
Existing component/
Software

25. DART: The DASH Runtime Interface
The DART API
Plain-C based interface
Follows the SPMD execution model
Defines Units and Teams
Defines a global memory abstraction
Provides a global pointer
Defines one-sided access operations (puts and gets)
Provides collective and pair-wise synchronization mechanisms
DASH Runtime (DART)
DASH C++ Template Library
DASH Application
Tools and Interfaces
Hardware: Network, Processor,
Memory, Storage
One-sided Communication
Substrate
MPI
GASnet
GASPI
ARMCI
DART API

26. Unit: individual participants in a DASH/DART program
Units and Teams
Unit: individual participants in a DASH/DART program
Unit ≈ process (MPI) ≈ thread (UPC) ≈ image (CAF)
Execution model follows the classical SPMD (single program multiple data) paradigm
Each unit has a global ID that remains unchanged during the execution
Team:
Ordered subset of units
Identified by an integer ID
DART_TEAM_ALL represents all units in a program
Units that are members of a team have a local ID with respect to that team

27. Communication: One-sided puts and gets
Blocking and non-blocking versions
Performance of blocking puts and gets closely matches MPI performance

28. DASH (C++ Template Library)
1D array as the basic data type
DASH follows a global-view approach, but local-view programming is supported too
Standard algorithms can be used but may not yield best performance
lbegin(), lend() allow iteration over local elements

29. Data Distribution Patterns
A Pattern controls the mapping of an index space onto units
A team can be specified (the default team is used otherwise)
No datatype is specified for a pattern
Patterns guarantee a similar mapping for different containers
Patterns can be used to specify parallel execution

30. Accessing and Working with Data in DASH (1)
GlobalPtr<T> abstraction that serves as the global iterator
GlobalRef<T> abstraction “reference to element in global memory” that is returned by subscript and iterator dereferencing

31. Accessing and Working with Data in DASH (2)
Range based for works on the global object per default
Proxy object can be used instead to access the local part of the data

32. Parallel programming is difficult
Summary
Parallel programming is difficult
PGAS is an interesting middle ground between message passing and shared memory programming with threads
Inherits the advantages of both but also shares some of the disadvantages – specifically race conditions
PGAS
Today mostly used when working with applications with irregular parallelism - random data accesses
UPC is the most widely used PGAS approach today
Co-array Fortran and other new PGAS languages
DASH and other C++ libraries
Thank you for your attention!