1. Multi-core Programming Threading Concepts

2. 2 Basics of VTune™ Performance Analyzer Topics A Generic Development Cycle Case Study: Prime Number Generation Common Performance Issues

3. 3 Threaded Programming Methodology What is Parallelism? Two or more processes or threads execute at the same time Parallelism for threading architectures – Multiple processes Communication through Inter-Process Communication (IPC) – Single process, multiple threads Communication through shared memory

4. 4 Threaded Programming Methodology n = number of processors T parallel = { (1-P) + P/n } T serial Speedup = T serial / T parallel Amdahl’s Law Describes the upper bound of parallel execution speedup Serial code limits speedup (1-P) P T serial (1-P) P/2 0.5 + 0.25 1.0/0.75 = 1.33 n = 2 n = ∞ ∞ P/ ∞ … 0.5 + 0.0 1.0/0.5 = 2.0 P - proportion of a computation where parallelization may occur..

5. 5 Threaded Programming Methodology Processes and Threads Modern operating systems load programs as processes Resource holder Execution A process starts executing at its entry point as a thread Threads can create other threads within the process – Each thread gets its own stack All threads within a process share code & data segments Code segment Data segment thread main() … thread Stack

6. 6 Threaded Programming Methodology Threads – Benefits & Risks Benefits – Increased performance and better resource utilization Even on single processor systems - for hiding latency and increasing throughput – IPC through shared memory is more efficient Risks – Increases complexity of the application – Difficult to debug (data races, deadlocks, etc.)

7. 7 Threaded Programming Methodology Commonly Encountered Questions with Threading Applications Where to thread? How long would it take to thread? How much re-design/effort is required? Is it worth threading a selected region? What should the expected speedup be? Will the performance meet expectations? Will it scale as more threads/data are added? Which threading model to use?

8. 8 Threaded Programming Methodology Prime Number Generation bool TestForPrime(int val) { // let’s start checking from 3 int limit, factor = 3; limit = (long)(sqrtf((float)val)+0.5f); while( (factor <= limit) && (val % factor) ) factor ++; return (factor > limit); } void FindPrimes(int start, int end) { int range = end - start + 1; for( int i = start; i <= end; i += 2 ) { if( TestForPrime(i) ) globalPrimes[gPrimesFound++] = i; ShowProgress(i, range); } ifactor 61 3 5 7 63 3 65 3 5 67 3 5 7 69 3 71 3 5 7 73 3 5 7 9 75 3 5 77 3 5 7 79 3 5 7 9

9. 9 Threaded Programming Methodology Development Methodology Analysis – Find computationally intense code Design (Introduce Threads) – Determine how to implement threading solution Debug for correctness – Detect any problems resulting from using threads Tune for performance – Achieve best parallel performance

10. 10 Threaded Programming Methodology Development Cycle Analysis –VTune™ Performance Analyzer Design (Introduce Threads) –Intel® Performance libraries: IPP and MKL –OpenMP* (Intel® Compiler) –Explicit threading (Win32*, Pthreads*) Debug for correctness –Intel® Thread Checker –Intel Debugger Tune for performance –Intel® Thread Profiler –VTune™ Performance Analyzer

11. 11 Threaded Programming Methodology Let’s use the project PrimeSingle for analysis PrimeSingle Usage:./PrimeSingle 1 1000000 Analysis - Sampling Use VTune Sampling to find hotspots in application bool TestForPrime(int val) { // let’s start checking from 3 int limit, factor = 3; limit = (long)(sqrtf((float)val)+0.5f); while( (factor <= limit) && (val % factor)) factor ++; return (factor > limit); } void FindPrimes(int start, int end) { // start is always odd int range = end - start + 1; for( int i = start; i <= end; i+= 2 ){ if( TestForPrime(i) ) globalPrimes[gPrimesFound++] = i; ShowProgress(i, range); } Identifies the time consuming regions

12. 12 Threaded Programming Methodology Analysis - Call Graph This is the level in the call tree where we need to thread Used to find proper level in the call-tree to thread

13. 13 Threaded Programming Methodology Analysis Where to thread? – FindPrimes() Is it worth threading a selected region? – Appears to have minimal dependencies – Appears to be data-parallel – Consumes over 95% of the run time Baseline measurement

14. 14 Threaded Programming Methodology Foster’s Design Methodology From Designing and Building Parallel Programs by Ian Foster Four Steps: – Partitioning Dividing computation and data – Communication Sharing data between computations – Agglomeration Grouping tasks to improve performance – Mapping Assigning tasks to processors/threads

15. 15 Threaded Programming Methodology Designing Threaded Programs Partition Partition – Divide problem into tasks Communicate Communicate – Determine amount and pattern of communication Agglomerate Agglomerate – Combine tasks Map Map – Assign agglomerated tasks to created threads The Problem Initial tasksCommunicationCombined Tasks Final Program

16. 16 Threaded Programming Methodology Parallel Programming Models Functional Decomposition – Task parallelism – Divide the computation, then associate the data – Independent tasks of the same problem Data Decomposition – Same operation performed on different data – Divide data into pieces, then associate computation

17. 17 Threaded Programming Methodology Decomposition Methods Functional Decomposition – Focusing on computations can reveal structure in a problem Grid reprinted with permission of Dr. Phu V. Luong, Coastal and Hydraulics Laboratory, ERDC Domain Decomposition Focus on largest or most frequently accessed data structure Data Parallelism Same operation applied to all data Atmosphere Model Ocean Model Land Surface Model Hydrology Model

18. 18 Threaded Programming Methodology Pipelined Decomposition Computation done in independent stages Functional decomposition – Threads are assigned stage to compute – Automobile assembly line Data decomposition – Thread processes all stages of single instance – One worker builds an entire car

19. 19 Threaded Programming Methodology LAME Pipeline Strategy Frame N Frame N + 1 Time Other N Prelude N Acoustics N Encoding N T 2 T 1 Acoustics N+1 Prelude N+1 Other N+1 Encoding N+1 Acoustics N+2 Prelude N+2 T 3 T 4 Prelude N+3 Hierarchical Barrier OtherPreludeAcousticsEncoding Frame Fetch next frame Frame characterization Set encode parameters Psycho Analysis FFT long/short Filter assemblage Apply filtering Noise Shaping Quantize & Count bits Add frame header Check correctness Write to disk

20. 20 Threaded Programming Methodology Design What is the expected benefit? How do you achieve this with the least effort? How long would it take to thread? How much re-design/effort is required? Rapid prototyping with OpenMP Speedup(2P) = 100/(96/2+4) = ~1.92X

21. 21 Threaded Programming Methodology OpenMP Fork-join parallelism: – Master thread spawns a team of threads as needed – Parallelism is added incrementally Sequential program evolves into a parallel program Parallel Regions Master Thread

22. 22 Threaded Programming Methodology Design #pragma omp parallel for for( int i = start; i <= end; i+= 2 ){ if( TestForPrime(i) ) globalPrimes[gPrimesFound++] = i; ShowProgress(i, range); } OpenMP Create threads here for this parallel region for Divide iterations of the for loop

23. 23 Threaded Programming Methodology Design What is the expected benefit? How do you achieve this with the least effort? How long would it take to thread? How much re-design/effort is required? Is this the best speedup possible? Speedup of 1.40X (less than 1.92X)

24. 24 Threaded Programming Methodology Debugging for Correctness Is this threaded implementation right? No! The answers are different each time …

25. 25 Threaded Programming Methodology Debugging for Correctness Intel® Thread Checker pinpoints notorious threading bugs like data races, stalls and deadlocks Intel ® Thread Checker VTune™ Performance Analyzer +DLLs (Instrumented) Binary Instrumentation Primes.exe (Instrumented) Runtime Data Collector threadchecker.thr (result file)

26. 26 Threaded Programming Methodology Thread Checker

27. 27 Threaded Programming Methodology Debugging for Correctness How much re-design/effort is required? How long would it take to thread? Thread Checker reported only 2 dependencies, so effort required should be low

28. 28 Threaded Programming Methodology Debugging for Correctness #pragma omp parallel for for( int i = start; i <= end; i+= 2 ){ if( TestForPrime(i) ) #pragma omp critical globalPrimes[gPrimesFound++] = i; ShowProgress(i, range); } #pragma omp critical { gProgress++; percentDone = (int)(gProgress/range *200.0f+0.5f) } Will create a critical section for this reference Will create a critical section for both these references

29. 29 Threaded Programming Methodology Correctness 1.33 Correct answer, but performance has slipped to ~1.33X Is this the best we can expect from this algorithm? No! From Amdahl’s Law, we expect speedup close to 1.9X

30. 30 Threaded Programming Methodology Common Performance Issues Parallel Overhead – Due to thread creation, scheduling … Synchronization – Excessive use of global data, contention for the same synchronization object Load Imbalance – Improper distribution of parallel work Granularity – No sufficient parallel work

31. 31 Threaded Programming Methodology Tuning for Performance Thread Profiler pinpoints performance bottlenecks in threaded applications Thread Profiler Thread Profiler VTune™ Performance Analyzer +DLL’s (Instrumented) Binary Instrumentation Primes.c Primes.exe (Instrumented) Runtime Data Collector Bistro.tp/guide.gvs (result file) Compiler Source Instrumentation Primes.exe /Qopenmp_profile

32. 32 Threaded Programming Methodology Thread Profiler for OpenMP

33. 33 Threaded Programming Methodology Thread Profiler for OpenMP Speedup Graph Estimates threading speedup and potential speedup – Based on Amdahl’s Law computation Gives upper and lower bound estimates

34. 34 Threaded Programming Methodology Thread Profiler for OpenMP serial parallel

35. 35 Threaded Programming Methodology Thread Profiler for OpenMP Thread 0Thread 1Thread 2Thread 3

36. 36 Threaded Programming Methodology Thread Profiler (for Explicit Threads)

37. 37 Threaded Programming Methodology Thread Profiler (for Explicit Threads) Why so many transitions?

38. 38 Threaded Programming Methodology Performance This implementation has implicit synchronization calls This limits scaling performance due to the resulting context switches Back to the design stage

39. Is that much contention expected? The algorithm has many more updates than the 10 needed for showing progress 39 Threaded Programming Methodology Performance void ShowProgress( int val, int range ) { int percentDone; gProgress++; percentDone = (int)((float)gProgress/(float)range*200.0f+0.5f); if( percentDone % 10 == 0 ) printf("\b\b\b\b%3d%", percentDone); } void ShowProgress( int val, int range ) { int percentDone; static int lastPercentDone = 0; #pragma omp critical { gProgress++; percentDone = (int)((float)gProgress/(float)range*200.0f+0.5f); } if( percentDone % 10 == 0 && lastPercentDone < percentDone / 10){ printf("\b\b\b\b%3d%", percentDone); lastPercentDone++; } This change should fix the contention issue

40. 40 Threaded Programming Methodology Design Goals – Eliminate the contention due to implicit synchronization 2.32X Speedup is 2.32X ! Is that right?

41. 41 Threaded Programming Methodology Performance Our original baseline measurement had the “flawed” progress update algorithm Is this the best we can expect from this algorithm? 1.40X Speedup is actually 1.40X (<<1.9X)!

42. 42 Threaded Programming Methodology Performance Re-visited Still have 62% of execution time in locks and synchronization

43. 43 Threaded Programming Methodology Performance Re-visited Let’s look at the OpenMP locks… void FindPrimes(int start, int end) { // start is always odd int range = end - start + 1; #pragma omp parallel for for( int i = start; i <= end; i += 2 ) { if( TestForPrime(i) ) #pragma omp critical globalPrimes[gPrimesFound++] = i; ShowProgress(i, range); } Lock is in a loop void FindPrimes(int start, int end) { // start is always odd int range = end - start + 1; #pragma omp parallel for for( int i = start; i <= end; i += 2 ) { if( TestForPrime(i) ) globalPrimes[InterlockedIncrement(&gPrimesFound)] = i; ShowProgress(i, range); }

44. 44 Threaded Programming Methodology Performance Re-visited Let’s look at the second lock void ShowProgress( int val, int range ) { int percentDone; static int lastPercentDone = 0; #pragma omp critical { gProgress++; percentDone = (int)((float)gProgress/(float)range*200.0f+0.5f); } if( percentDone % 10 == 0 && lastPercentDone < percentDone / 10){ printf("\b\b\b\b%3d%", percentDone); lastPercentDone++; } This lock is also being called within a loop void ShowProgress( int val, int range ) { long percentDone, localProgress; static int lastPercentDone = 0; localProgress = InterlockedIncrement(&gProgress); percentDone = (int)((float)localProgress/(float)range*200.0f+0.5f); if( percentDone % 10 == 0 && lastPercentDone < percentDone / 10){ printf("\b\b\b\b%3d%", percentDone); lastPercentDone++; }

45. 45 Thread Profiler for OpenMP Thread 0 342 factors to test 116747 Thread 1 612 factors to test 373553 Thread 2 789 factors to test 623759 Thread 3 934 factors to test 873913 500000 250000 750000 1000000

46. 46 Threaded Programming Methodology Fixing the Load Imbalance Distribute the work more evenly void FindPrimes(int start, int end) { // start is always odd int range = end - start + 1; #pragma omp parallel for schedule(static, 8) for( int i = start; i <= end; i += 2 ) { if( TestForPrime(i) ) globalPrimes[InterlockedIncrement(&gPrimesFound)] = i; ShowProgress(i, range); } 1.68X Speedup achieved is 1.68X

47. 47 Threaded Programming Methodology Final Thread Profiler Run 1.80X Speedup achieved is 1.80X

48. 48 Threaded Programming Methodology Comparative Analysis Threading applications require multiple iterations of going through the software development cycle

49. 49 Threaded Programming Methodology Threading Methodology What’s Been Covered Four step development cycle for writing threaded code from serial and the Intel® tools that support each step – Analysis – Design (Introduce Threads) – Debug for correctness – Tune for performance Threading applications require multiple iterations of designing, debugging and performance tuning steps Use tools to improve productivity