Presentation is loading. Please wait.

Presentation is loading. Please wait.

IBM ATS Deep Computing © 2007 IBM Corporation Compiler Optimization HPC Workshop – University of Kentucky May 9, 2007 – May 10, 2007 Andrew Komornicki,

Published byIsaac Jeffrey Hodge Modified over 9 years ago

Similar presentations

Presentation on theme: "IBM ATS Deep Computing © 2007 IBM Corporation Compiler Optimization HPC Workshop – University of Kentucky May 9, 2007 – May 10, 2007 Andrew Komornicki,"— Presentation transcript:

1 IBM ATS Deep Computing © 2007 IBM Corporation Compiler Optimization HPC Workshop – University of Kentucky May 9, 2007 – May 10, 2007 Andrew Komornicki, Ph. D. Balaji Veeraraghavan, Ph. D.

2 IBM ATS Deep Computing © 2007 IBM Corporation Agenda  Code Optimization Examples  Outer loop unrolling  Inner loop unrolling  Stride one memory access  Cache blocking  Hardware prefetch  Memory store _dcbz  Optimization for arithmetic operations Divide, SQRT, etc

3 IBM ATS Deep Computing © 2007 IBM Corporation Understanding Performance  Fused floating multiply add functional units  Latency  Balance  Memory Access  Stride  Latency  Bandwidth

4 IBM ATS Deep Computing © 2007 IBM Corporation FMA Functional Unit  Multiply and Add  Fused  6 clock period latency D = A+B*C

5 IBM ATS Deep Computing © 2007 IBM Corporation Programming Concerns  Superscalar design  Concurrency: Branches Loop control Procedure calls Nested “if” statements  Program “control” is very efficient  Cache based microprocessor  Critical resource: memory bandwidth Tactics: >Increase Computational Intensity >Exploit "prefetch"

6 IBM ATS Deep Computing © 2007 IBM Corporation Strategies for Optimization  Exploit multiple functional units  Exploit FMA capability  Expose load/store "streams"  Use optimized libraries  Pipelined operations

7 IBM ATS Deep Computing © 2007 IBM Corporation Tactics for Optimization  Cache reuse  Blocking  Unit stride  Use entire loaded cache lines  Limit range of indirect addressing  Sort indirect addresses  Increase computational intensity of the loops  Ratio of flop's to byte's  Load streams

8 IBM ATS Deep Computing © 2007 IBM Corporation Computational Intensity  Ratio of floating point operations per memory references (loads and stores)  Higher is better  Example: do I=1,n A(I) = A(I) +B(I)*s+C(I) enddo  Loads and stores: 3+1  Floating point operations: 3  Computational intensity: 3/4

9 IBM ATS Deep Computing © 2007 IBM Corporation Loop Unrolling Strategy: Increase Computational Intensity  Find variable which is constant with respect to outer loop  Unroll so that this variable is loaded once but used multiple times  Unroll outer loop  Minimizes load/stores

10 IBM ATS Deep Computing © 2007 IBM Corporation Example 1: Outer Loop Unroll  2 flops / 2 loads  Comp. Int.: 1  8 flops / 5 Loads  Comp. Int.: 1.6 DO I = 1, N DO J = 1, N s = s +X(J)*A(J,I) END DO DO I= 1, N, 4 DO J = 1, N s = s +X(J)*A(J,I+0) +X(J)*A(J,I+1) +X(J)*A(J,I+2) +X(J)*A(J,I+3) END DO Unroll

11 IBM ATS Deep Computing © 2007 IBM Corporation Outer Loop Unroll Test Best hand unroll Best compiler unroll Best combined unroll Theoretical peak = 1.3X4 = 5200 Gflop/s

12 IBM ATS Deep Computing © 2007 IBM Corporation Loop Unroll Analysis  Hand unroll:  Unroll up to 8 times Exploit 8 prefetch streams  Compiler unroll:  Unrolls up to 4 times “Near” optimal performance  Combines inner and outer loop unrolling  Goals of this example  Demonstrate how compiler option works  Compiler can do fairly good job on loop unrolling

13 IBM ATS Deep Computing © 2007 IBM Corporation Inner Loop Unrolling Strategies  Benefit of Inner loop unroll:  Reduces data dependency  Eliminate intermediate loads and stores  Expose functional units  Expose registers  Examples: Linear recurrences Simple loops –Single operation

14 IBM ATS Deep Computing © 2007 IBM Corporation Example 2: Inner Loop Unroll: Turning Data Dependency into Data Reuse  Enable data reuse  Eliminate data dependency (half)  Eliminate intermediate loads and stores  Compiler will do some of this at -O3 and higher do i=2,n-1 a(i+1) = a(i)*s1 + a(i-1)*s2 end do do i=2,n-2,2 a(i+1) = a(i )*s1 + a(i-1)*s2 a(i+2) = a(i+1)*s1 + a(i )*s2 end do Unroll a(i) and a(i+1) used twice

15 IBM ATS Deep Computing © 2007 IBM Corporation Inner Loop Unroll: Dependencies POWER4 1.3 Best hand unroll Best compiler unroll Best combined unroll By Hand By compiler

16 IBM ATS Deep Computing © 2007 IBM Corporation Inner Loop Unroll: Dependencies  Compiler unrolling helps  Does not help manually unrolled loops Conflicts  Turn off compiler unrolling if manually unrolled

17 IBM ATS Deep Computing © 2007 IBM Corporation Example 3: Inner Loop Unroll: Registers and Functional Units  Expose functional units  Expose registers  Compiler will do some of this at -O3 and higher do j=1,n do i=1,m sum = sum + X(i)*A(i,j) end do do j=1,n do i=1,m,2 sum = sum + X(i)*A(i,j) & + X(i+1)*A(i+1,j) end do Unroll

18 IBM ATS Deep Computing © 2007 IBM Corporation Inner Loop Unroll: Registers and Functional Units POWER4 1.45 Best hand unroll Best compiler unroll Best combined unroll By Hand

19 IBM ATS Deep Computing © 2007 IBM Corporation Example 4 Outer Loop Unrolling Strategies  Expose prefetch streams  Up to 8 streams do j=1,n do i=1,m sum = sum + X(i)*A(i,j) end do do j=1,n,2 do i=1,m sum = sum + X(i)*A(i,j) & + X(i)*A(i,j+1) end do Unroll

20 IBM ATS Deep Computing © 2007 IBM Corporation Outer Loop Unroll: Streams POWER4 1.45 Best hand unroll Best compiler unrollBest combined unroll

21 IBM ATS Deep Computing © 2007 IBM Corporation Outer Loop Unroll: Streams  Compiler does a “fair” job of outer loop unrolling  Manual unrolling can outperform compiler unrolling

22 IBM ATS Deep Computing © 2007 IBM Corporation Summary: Turn on Loop Unrolling  Clear performance benefit  Consider both inner loop and outer loop unrolling  Turn on the compiler options and let it do the work first -O3 –qhot –qunroll=yes -qunroll=auto is default, but it is not agressive  If you really want to do it by hand  Then you may want to turn off compiler unroll by using -qnounroll

23 IBM ATS Deep Computing © 2007 IBM Corporation New Topic: Maximize Cache Memory  Caches are fast and more expensive memories. Let’s take full advantage of them !!  A cache line is the minimum unit of data transfer.

24 IBM ATS Deep Computing © 2007 IBM Corporation Memory Strided Memory Access for (i=0;i<n;i+=2) sum+=a[i];  Strided memory access uses only a portion of a cache line  Reduced efficiency  Example, stride two memory access 2 4 6 8 10 12 14 16 2 nd level cache line

25 IBM ATS Deep Computing © 2007 IBM Corporation Strides  Power5 Cache line size is 128 bytes  Double precision:16 words  Single precision:32 words Bandwidth actually used StrideSingleDouble 11x 2½½ 4¼¼ 81/8 161/16 321/321/16 641/321/16 for (i=0;i<n;i+=6) sum+=a[i]; for (i=n;i>0;i+=-6) sum+=a[i]; POWER4 1.3 GHz Stride

26 IBM ATS Deep Computing © 2007 IBM Corporation Correcting Strides  Interleave code  Example:  Real and Imaginary part arithmetic do i=1,n … AIMAG(Z(i)) end do do i=1,n …REAL(Z(i)) end do do i=1,n … AIMAG(Z(i)) … …REAL(Z(i)) end do Only ½ of cache bandwidth is used Full cache bandwidth is used

27 IBM ATS Deep Computing © 2007 IBM Corporation Example 5: Array Index  Perform loop Interchange do i=1,n do j=1,m sum=sum+A(i,j) end do Interchange do j=1,m do i=1,n sum=sum+A(i,j) end do For optimal cache line usage, a Fortran code should make the LEFTMOST array index equal the inner most loop index. A C code should do the opposite: the RIGHTMOST array index should equal the innermost loop index.

28 IBM ATS Deep Computing © 2007 IBM Corporation Correcting Strides: Loop Interchange 1.45 GHz POWER4 Loop interchange is easy, so let’s do it by hand hand unroll compiler unroll

29 IBM ATS Deep Computing © 2007 IBM Corporation TLB - Translation Lookaside Buffer  A buffer (or cache) in a CPU to improve the speed of virtual address translationCPU  Contains parts of the page table which translates from virtual into physical addresses.page tablevirtual physical addresses  Has a fixed number of entries  A TLB miss occurs when the page table data needed for translation is not in TLB.  Such translation will proceed via the page table  Take several more cycles to complete – particularly if the translation tables are swapped out into secondary storage  Memory access with large stride = random memory access  Can easily cause TLB misses  Also has very low cache line reuse

30 IBM ATS Deep Computing © 2007 IBM Corporation Negative Effect of Large Strides - TLB Misses POWER4 1.3 GHz

31 IBM ATS Deep Computing © 2007 IBM Corporation Working Set Size  Roughly equals the memory used in a loop  Reduce spanning size of random memory accesses  More MPI tasks usually helps

32 IBM ATS Deep Computing © 2007 IBM Corporation Blocking  Common technique used in linear algebraic operations such as BLAS  Similar to unrolling  Utilize cache lines  Linear Algebra Typically 96-256

33 IBM ATS Deep Computing © 2007 IBM Corporation Blocking Do j=1,jmax Do i=1,imax.... End do Do j=1,jmax,jblock Do i=1,imax,iblock do one block of work.... End do

34 IBM ATS Deep Computing © 2007 IBM Corporation Blocking Example: Transpose  Especially useful for bad strides do i = 1,n do j = 1,m B(j,i) = A(i,j) end do do j1 = 1,n-nb+1,nb j2 = min(j1+nb-1,n) do i1 = 1,m-nb+1,nb i2 = min(i1+nb-1,m) do i = i1, i2 do j = j1, j2 B(j,i) = A(i,j) end do Blocking

35 IBM ATS Deep Computing © 2007 IBM Corporation Blocking Example: Transpose POWER4 1.3 GHz

36 IBM ATS Deep Computing © 2007 IBM Corporation Hardware Prefetch  Detects adjacent cache line references  Forward and backward  Up to eight concurrent streams  Prefetches up to two lines ahead per stream  Twelve prefetch filter queues prevents rolling  No prefetch on store misses  (when a store instruction causes a cache line miss)

37 IBM ATS Deep Computing © 2007 IBM Corporation Prefetch: Stride Pattern Recognition  Upon a cache miss:  Biased guess is made as to the direction of that stream  Guess is based upon where in the cache line the address associated with that miss occurred  If it is in the first 3/4, then the direction is guessed as ascending  If in the last 1/4, the direction is guessed descending

38 IBM ATS Deep Computing © 2007 IBM Corporation Memory Bandwidth  Bandwidth is proportional to number of streams  Streams are roughly the number of right hand side arrays  Up to eight streams

39 IBM ATS Deep Computing © 2007 IBM Corporation Exploiting Prefetch  Merge Loops  Strategy  Combine loops to get up to 8 right hand sides for (j=1; j<= n; j++) A[j] = A[j-1]+B[j] for (j=1; j<= n; j++) D[j] = D[j+1]+C[j]*s for (j=1; j<= n; j++) { A[j] = A[j-1]+B[j] D[j] = D[j+1]+C[j]*s } merge

40 IBM ATS Deep Computing © 2007 IBM Corporation Folding (Similar to Loop Unrolling)  Fold loop to increase number of streams:  Strategy  Fold loops to get up to 4 times do i = 1,n sum = sum +A(i) end do do i = 1,n/4 sum = sum +A(i ) + A(i+1*n/4) + A(i+2*n/4) + A(i+3*n/4) end do

41 IBM ATS Deep Computing © 2007 IBM Corporation Folding Example POWER4 1.3 GHz 0 500 1000 1500 2000 2500 3000 3500 4000 Mflop/s 0246812 Folds

42 IBM ATS Deep Computing © 2007 IBM Corporation Effect of Precision  Available floating point formats:  real (kind=4)  real (kind=8)  real (kind=16)  Advantage of smaller data types:  Require less bandwidth  More effective cache use REAL*8 A,pi,e … do i=1,n A(i) = pi*A(i) + e end do REAL*4 A,pi,e … do i=1,n A(i) = pi*A(i) + e end do

43 IBM ATS Deep Computing © 2007 IBM Corporation Effect of Precision (word size) POWER4 1.3 GHz Array Size

44 IBM ATS Deep Computing © 2007 IBM Corporation Divide and Sqrt  POWER special functions:  Divide  Sqrt  Use FMA functional unit  2 simultaneous divide or sqrt (or rsqrt)  NOT pipelined Cycles Instruction Single Precision Double Precision Fma66 Fdiv2228 Fsqrt2835

45 IBM ATS Deep Computing © 2007 IBM Corporation Hardware DIV, SQRT, RSQRT POWER4 1.3 GHz

46 IBM ATS Deep Computing © 2007 IBM Corporation Intrinsic Function Vectorization (Example: Program SPPM) do i = -nbdy+3,n+nbdy-1 prl = qrprl(i) pll = qrmrl(i) pavg = vtmp1(i) wllfac(i) = 5*gammp1*pavg + gamma * pll wrlfac(i) = 5*gammp1 * pavg +gamma *prl hrholl = rho(1,i-1) hrhorl = rho(1,i) wll(i) = 1/sqrt(hrholl * wllfac(i)) wrl(i) = 1/sqrt(hrhorl * wrlfac(i)) end do Take these out of loop to make vectors

47 IBM ATS Deep Computing © 2007 IBM Corporation Intrinsic Function Vectorization allocate(t1,n+2*nbdy-3) allocate(t2,n+2*nbdy-3) do i = -nbdy+3,n+nbdy-1 prl = qrprl(i)... t1(i) =hrholl * wllfac(i) t2(i) =hrhorl * wrlfac(i) end do call __vrsqrt(t1,wrl,n+2*nbdy-3) call __vrsqrt(t2,wll,n+2*nbdy-3)

48 IBM ATS Deep Computing © 2007 IBM Corporation Vectorization Analysis  Dependencies  Compiler overhead:  Generate (malloc) local temporary arrays  Extra memory traffic  Moderate vector lengths required RECSQRTRSQRT Crossover Length 2080 ??25?? N½N½ 452530

49 IBM ATS Deep Computing © 2007 IBM Corporation Function Timings Function Hardware Clocks Hardware Rate [Mop/s] Pipelined (Vec’ed) Clocks Pipelined Rate [Mop/s] Recip.328120130 SQRT36693184 RSQRT66392990 EXP200133283 LOG28893477

50 IBM ATS Deep Computing © 2007 IBM Corporation SQRT  POWER4/POWER5 have hardware SQRT available  Default -qarch=com uses software library  Use: -qarch=pwr4

51 IBM ATS Deep Computing © 2007 IBM Corporation SQRT  Hardware SQRT is 5x faster  -qhot generates "vector sqrt"

52 IBM ATS Deep Computing © 2007 IBM Corporation Divide  IEEE divide specifies actual divide  Do not use multiply by reciprocal (default)  Optimize with -O3 do i=1,n B(i) = A(i)/s end do rs=1/s do i=1,n B(i) = A(i)*rs end do

53 IBM ATS Deep Computing © 2007 IBM Corporation DIVIDE POWER4 1.3 GHz

54 IBM ATS Deep Computing © 2007 IBM Corporation Vector Functions  -qhot generates "vector" interface to intrinsic functions  Monitor with "-qreport=hotlist" do i=1,n B(i) = func(A(i)) end do call __vfunc(B,A,n) -qhot

55 IBM ATS Deep Computing © 2007 IBM Corporation Vector Functions POWER4 1.3 GHz

56 IBM ATS Deep Computing © 2007 IBM Corporation Power Function  Computing power function: a**b  if ( b < 10 and has integer value at compile time) Use unrolling and successive multiply  else... __pow(...) Test case: do i=1,n A(i)=B(i)**b end do

57 IBM ATS Deep Computing © 2007 IBM Corporation Power Function POWER4 1.3 GHz

58 IBM ATS Deep Computing © 2007 IBM Corporation Power Function  Compiler can transform integer power to multiply's  Real to Real power (r r ) is expensive  Real to Integer (r I ) is less expensive

59 IBM ATS Deep Computing © 2007 IBM Corporation Effect of Compiler Option Levels POWER4 1.3 GHz

60 IBM ATS Deep Computing © 2007 IBM Corporation Effect of 32- and 64- bit Addressing  64-bit address mode enable use of 64-bit integer arithmetic  Integer Arithmetic, especially (kind=8), is much faster with - q64 FortranC, C++ Address Mode ComputationInteger*4Integer*8long -q324 bytes 8 bytes4 bytes8 bytes -q644 or 8 bytes4 bytes8 bytes

61 IBM ATS Deep Computing © 2007 IBM Corporation 32-bit vs. 64-bit Mode Integer Arithmetic POWER4 1.3 GHz

62 IBM ATS Deep Computing © 2007 IBM Corporation Effect of 32-bit Floating Point  Faster because of bandwidth  Arithmetic operations are same speed as 64-bit  More efficient use of cache

63 IBM ATS Deep Computing © 2007 IBM Corporation Use of Library Routines: Level 1  Single level loop constructs  Examples BLAS 1 Memory Copy  Prefer compiler generated code Better than: –libc.a >memcopy >bcopy >memmove –ESSL >dcopy

64 IBM ATS Deep Computing © 2007 IBM Corporation Memory Copy Routines POWER4 1.3 GHz

65 IBM ATS Deep Computing © 2007 IBM Corporation Use of Library Routines: Level 2  Two level loop constructs  Examples  Matrix Vector Multiply DGEMV  Prefer ESSL or liblapack

66 IBM ATS Deep Computing © 2007 IBM Corporation DGEMV POWER4 1.3 GHz

67 IBM ATS Deep Computing © 2007 IBM Corporation Use of Library Routines: Level 3  Three level loop constructs  Examples  Matrix Matrix Multiply  DGEMM  Prefer ESSL or liblapack

68 IBM ATS Deep Computing © 2007 IBM Corporation DGEMM POWER4 1.3 GHz

69 IBM ATS Deep Computing © 2007 IBM Corporation Subscript Reorder  Reference on second or higher subscript has bad strides  Reduced performance  Fixed by:  -qhot can interchange some loops  !IBM* SUBSCRIPTORDER(A(2,1)) do i=1,n do j=1,m s = s + A(i,j) end do

70 IBM ATS Deep Computing © 2007 IBM Corporation !IBM* SUBSCRIPTORDER POWER4 1.3 GHz

71 IBM ATS Deep Computing © 2007 IBM Corporation Summary  Use unrolling of outer loops  Compiler “might” do this  Loop at number of prefetch streams  8 is optimal  Use –q64  Enables 64 bit integer arithmetic  Use –qarch=auto  Enables divide and square root instructions

72 IBM ATS Deep Computing © 2007 IBM Corporation  Questions ?

Download ppt "IBM ATS Deep Computing © 2007 IBM Corporation Compiler Optimization HPC Workshop – University of Kentucky May 9, 2007 – May 10, 2007 Andrew Komornicki,"

Similar presentations

Part IV: Memory Management

Part IV: Memory Management
Compiler Support for Superscalar Processors. Loop Unrolling Assumption: Standard five stage pipeline Empty cycles between instructions before the result.

Compiler Support for Superscalar Processors. Loop Unrolling Assumption: Standard five stage pipeline Empty cycles between instructions before the result.
DSPs Vs General Purpose Microprocessors

DSPs Vs General Purpose Microprocessors
Computer Organization and Architecture

Computer Organization and Architecture
Optimizing Compilers for Modern Architectures Allen and Kennedy, Chapter 13 Compiling Array Assignments.

Optimizing Compilers for Modern Architectures Allen and Kennedy, Chapter 13 Compiling Array Assignments.
School of EECS, Peking University “Advanced Compiler Techniques” (Fall 2011) Parallelism & Locality Optimization.

School of EECS, Peking University “Advanced Compiler Techniques” (Fall 2011) Parallelism & Locality Optimization.
Chapter 3 Instruction Set Architecture Advanced Computer Architecture COE 501.

Chapter 3 Instruction Set Architecture Advanced Computer Architecture COE 501.
Multi-Level Caches Vittorio Zaccaria. Preview What you have seen: Data organization, Associativity, Cache size Policies -- how to manage the data once.

Multi-Level Caches Vittorio Zaccaria. Preview What you have seen: Data organization, Associativity, Cache size Policies -- how to manage the data once.
EECC551 - Shaaban #1 Fall 2003 lec#3 9-16-2003 Pipelining and Exploiting Instruction-Level Parallelism (ILP) Pipelining increases performance by overlapping.

EECC551 - Shaaban #1 Fall 2003 lec# Pipelining and Exploiting Instruction-Level Parallelism (ILP) Pipelining increases performance by overlapping.
Processor Overview Features Designed for consumer and wireless products RISC Processor with Harvard Architecture Vector Floating Point coprocessor Branch.

Processor Overview Features Designed for consumer and wireless products RISC Processor with Harvard Architecture Vector Floating Point coprocessor Branch.
Vector Processing. Vector Processors Combine vector operands (inputs) element by element to produce an output vector. Typical array-oriented operations.

Vector Processing. Vector Processors Combine vector operands (inputs) element by element to produce an output vector. Typical array-oriented operations.
Parallell Processing Systems1 Chapter 4 Vector Processors.

Parallell Processing Systems1 Chapter 4 Vector Processors.
Computer Organization CS224 Fall 2012 Lesson 44. Virtual Memory  Use main memory as a “cache” for secondary (disk) storage l Managed jointly by CPU hardware.

Computer Organization CS224 Fall 2012 Lesson 44. Virtual Memory  Use main memory as a “cache” for secondary (disk) storage l Managed jointly by CPU hardware.
Lecture 34: Chapter 5 Today’s topic –Virtual Memories 1.

Lecture 34: Chapter 5 Today’s topic –Virtual Memories 1.
Compiler Challenges for High Performance Architectures

Compiler Challenges for High Performance Architectures
The Memory Hierarchy (Lectures #24) ECE 445 – Computer Organization The slides included herein were taken from the materials accompanying Computer Organization.

The Memory Hierarchy (Lectures #24) ECE 445 – Computer Organization The slides included herein were taken from the materials accompanying Computer Organization.
Data Locality CS 524 – High-Performance Computing.

Data Locality CS 524 – High-Performance Computing.
S.1 Review: The Memory Hierarchy Increasing distance from the processor in access time L1$ L2$ Main Memory Secondary Memory Processor (Relative) size of.

S.1 Review: The Memory Hierarchy Increasing distance from the processor in access time L1$ L2$ Main Memory Secondary Memory Processor (Relative) size of.
Computational Astrophysics: Methodology 1.Identify astrophysical problem 2.Write down corresponding equations 3.Identify numerical algorithm 4.Find a computer.

Computational Astrophysics: Methodology 1.Identify astrophysical problem 2.Write down corresponding equations 3.Identify numerical algorithm 4.Find a computer.
Cache effective mergesort and quicksort Nir Zepkowitz Based on: “Improving Memory Performance of Sorting Algorithms” by Li Xiao, Xiaodong Zhang, Stefan.

Cache effective mergesort and quicksort Nir Zepkowitz Based on: “Improving Memory Performance of Sorting Algorithms” by Li Xiao, Xiaodong Zhang, Stefan.

Similar presentations

Ads by Google