1. Parallel Programming pt.1
DaMiri young
Unt High Performance Computing
October 5, 2017

2. Overview Goal: working knowledge of parallel programming concepts.
Outline:
Concepts
Models
Examples
Challenges
Q&A/Conclusion

3. What is COMPUTING?
Computing or scientific computing is the use of advanced computing capabilities to understand and solve complex problems. At its core, it is the development of models and simulations to understand systems in a tractable way. Common computing paradigms:
Serial
Parallel

4. What is Serial COMPUTING?
Most research software is written for serial computation:
Problem is decomposed into a discrete set of instructions
Instructions are then executed sequentially one after another on a single processor
Only one instruction is in execution at a time

5. Serial COMPUTING…
Most research software is written for serial computation:
Problem is decomposed into a discrete set of instructions
Instructions are then executed sequentially one after another on a single processor
Only one instruction is in execution at a time

6. What is PARALLel COMPUTING?
Parallel computing is concurrent use of multiple compute resources to solve a computational problem:
Problem is decomposed into a discrete units that can be solved concurrently
Units are then further reduced to a set of instructions
Instructions are then executed simultaneously on multiple compute units/processors
An overall control/coordination mechanism is employed
Caveats:
Problem must be decomposable into discrete parts that can be solved concurrently
Must be able to execute multiple program instructions at a time
Problem must be solvable in less time with multiple compute resources than with a single resource

7. PARALLel COMPUTING…

8. PARALLel COMPUTING…

9. PARALLel COMPUTING…
Consider a robot moving blocks from location A to location B
It takes x mins / block and there 25 blocks so the total time required is 25x
How do we speed up moving 25 blocks from A -> B

10. PARALLel COMPUTING…
Get more robots moving blocks from location A to location B
I.e., 5 robots = 5X speedup, so total time should be 5x

11. Why Use PARALLEL COMPUTING?
Multiple processors in parallel in general solve problems quicker than single processors in serial!
A contrived example:
Suppose it takes 30 minutes to multiply an n x n matrix
on 1 processor. It is parallelized to run on 16 processors
(cores). Naively we might expect it to take ~1.8 minutes.

12. Why Use PARALLEL COMPUTING?
More efficient use of underlying hardware
Provide concurrency
Save money/time
Solve larger more complex problems
Utilization of remote resources

13. PARALLEL COMPUTING ARchiTecture…
Closely follows Von Neumann Architecture a.k.a stored program computer:
4 main components:
Memory
Control Unit
Arithmetic Unit
Input/Output

14. PARALLEL COMPUTING ARchiTecture…
Random Access Memory (RAM) – used to store both program instructions and data
Program instructions are coded data that instruct computer
Data is information that will be utilized by the program
Control Unit
Fetches instructions/data from memory
Decodes instructions
Sequentially coordinates operations
Arithmetic Unit
Performs basic arithmetic operations
Input/Output – basic interface to the human operator

15. PARALLEL COMPUTING ARchiTecture…
Flynn’s Taxonomy - distinguishes multi-processor computer architectures according to how they can be classified along the two independent dimensions of Instruction Stream and Data Stream. Each of these dimensions can have only one of two possible states: Single or Multiple.
Single Instruction, Single Data (SISD)
Single Instruction, Multiple Data (SIMD)
Multiple Instruction, Single Data (MISD)
Multiple Instruction, Multiple Data (MIMD)

16. PARALLEL ProgRamming Models
Parallel programming models are considered an abstracted layer above hardware and memory architecture.
Common models:
Shared Memory (without threads)
Threads
Distributed Memory / Message Passing
Data Parallel
Hybrid
Single Program Multiple Data (SPMD)
Multiple Program Multiple Data (MPMD)

17. PARALLEL ProgRamming Models…
Shared Memory (without threads)
This is perhaps the simplest parallel programming model.
In this programming model, processes/tasks share a common address space, which they read and write to asynchronously.
Various mechanisms such as locks / semaphores are used to control access to the shared memory, resolve contentions and to prevent race conditions and deadlocks.
An important disadvantage in terms of performance is that it becomes more difficult to understand and manage data locality: Keeping data local to the process that works on it conserves memory accesses, cache refreshes and bus traffic that occurs when multiple processes use the same data.
Unfortunately, controlling data locality is hard to understand and may be beyond the control of the average user.

18. PARALLEL ProgRamming Models…
Shared Memory (without threads) Implementation(s):
On stand-alone shared memory machines, native operating systems, compilers and/or hardware provide support for shared memory programming. For example, the POSIX standard provides an API for using shared memory, and UNIX provides shared memory segments (shmget, shmat, shmctl, etc).
On distributed memory machines, memory is physically distributed across a network of machines, but made global through specialized hardware and software. A variety of SHMEM implementations are available:

19. PARALLEL ProgRamming Models…
Thread
This programming model is a type of shared memory programming.
In the threaded model of parallel programming, a single "heavy weight" process can have multiple "light weight", concurrent execution paths.
programmer is responsible for determining the parallelism (although compilers can sometimes help).

20. PARALLEL ProgRamming Models…
Thread Implementation(s)
POSIX Threads
C Language only.
Part of Unix/Linux operating systems
Library based
Commonly referred to as Pthreads.
Very explicit parallelism; requires significant programmer attention to detail.
Others:
Microsoft threads
Java, Python threads
CUDA threads for GPUs
OpenMP
Compiler directive based
Portable / multi-platform, including Unix and Windows platforms
Available in C/C++ and Fortran implementations
Can be very easy and simple to use - provides for "incremental parallelism".
Can begin with serial code.
More Info:
POSIX Threads tutorial: computing.llnl.gov/tutorials/pthreads
OpenMP tutorial: computing.llnl.gov/tutorials/openMP

21. PARALLEL ProgRamming Models…
Distributed Memory / Message Passing
Multiple tasks can reside on the same physical machine and/or across an arbitrary number of machines.
A set of tasks that use their own local memory during computation.
Tasks exchange data through communications by sending and receiving messages.
Data transfer usually requires cooperative operations to be performed by each process. For example, a send operation must have a matching receive operation.

22. PARALLEL ProgRamming Models…
Distributed Memory / Message Passing Implementation(s):
Message Passing Interface (MPI-1) was released in 1994.
MPI-2 was released in 1996
MPI-3 in 2012.
More Info:
All MPI specifications are available on the web at forum.org/docs/.
MPI tutorial: computing.llnl.gov/tutorials/mpi

23. PARALLEL PROGRAMMING MINDSET
Understand BOTH the problem and the program – Analyze the problem you’re trying to solve including existing code. BE THE PROGRAM
Investigate other algorithms if possible. This may be the single most important consideration when designing a parallel application.
Determine whether or not the problem can be parallelized and at what cost.
Ex/ Calculate the potential energy for each of several thousand independent conformations of a molecule.
> Can be parallelized, each of the molecular conformations is independently determinable.
Ex/ Fibonacci series: F(n) = F(n-1) + F(n-2) = (0,1,1,2,3,5,8,13,21,… etc)
> Not easy to parallelized , F(n) value uses both F(n-1) and F(n-2) which must be computed. Data dependence issue

24. PARALLEL PROGRAMMING MINDSET
Identify the program's hotspots (Be the program):
Know where most of the real work is being done. The majority of scientific and technical programs usually accomplish most of their work in a few places.
Focus on parallelizing the hotspots and ignore those sections of the program that account for little CPU usage.
Profilers and performance analysis tools can help
Identify bottlenecks in the program:
Are there areas that are disproportionately slow, or cause parallelizable work to halt or be deferred? For example, I/O is usually something that slows a program down.
May be possible to restructure the program or use a different algorithm to reduce or eliminate unnecessary slow areas
Identify inhibitors to parallelism. One common class of inhibitor is data dependence, as demonstrated by the Fibonacci sequence above.
Use optimized third party parallel software and highly optimized math libraries available from leading vendors

25. PARALLEL PROGRAMMING MINDSET
Partitioning - breaking the problem into discrete "chunks" of work that can be distributed to multiple tasks. This is known as decomposition or partitioning. Types:
Domain Decomposition: In this type of partitioning, the data associated with a problem is decomposed. Each parallel task then works on a portion of the data.
Functional Decomposition: In this approach, the focus is on the computation that is to be performed rather than on the data manipulated by the computation. The problem is decomposed according to the work that must be done. Each task then performs a portion of the overall work.

26. PARAllEL Programming EXAMPLEs
Ex1 – Serial/Parallel (Non-Shared Memory) Ex2 – Threaded (Shared Memory) Ex3 – Distributed

27. PARAllEL Programming Challenges
Cost – Programmers Time, Coding, Design, Debugging, Tuning, etc.
Efficiency – Performance!!
Hidden Serialization – Intrinsic, Internal Serializations, I/O Hotspots,
Parallel Overhead – Communication, Latency
Load balancing – Even distribution of workload
Synchronization Issues - deadlocks

28. PARAllEL Programming Challenges
Speedup - defined as the ratio of the serial runtime of the best sequential algorithm for solving a problem to the time taken by the parallel algorithm to solve the same problem on p processors. Let Nproc be the number of parallel processes Speedup(NProc) =Time used by best serial program/Time used by parallel program

29. PARAllEL Programming Challenges
Efficiency – defined as the ratio of speedup to the number of processors. Measures the fraction of time for which a processor is usefully utilized. Efficiency(NProc) = Speedup(NProc)/NProc

30. PARAllEL Programming Challenges
Amdahl’s Law - states that potential program speedup is defined by the fraction of code (P) that can be parallelized:
If none of the code can be parallelized, P = 0 and the speedup = 1 (no speedup).
If all of the code is parallelized, P = 1 and the speedup is infinite (in theory).
If 50% of the code can be parallelized, maximum speedup = 2, meaning the code will run twice as fast.
Introducing the number of processors performing the parallel fraction of work, the relationship can be modeled by:
Speedup = ( 1 / P + S ) /N
Where p = parallel fraction, N=number of processors and S = serial fraction

31. Q & A ?

32. …
“You can spend a lifetime getting 95% of your code to be parallel, and never achieve better than 20x speedup no matter how many processors you throw at it! “ –
Unknown