1. March 14, 20021 CMPUT680 - Winter 2006 Topic C: Loop Fusion Kit Barton www.cs.ualberta.ca/~cbarton

2. March 14, 20022 Outline Definition of loop fusion Basic concepts Prerequisites of loop fusion A loop fusion algorithm Example

3. March 14, 20023 Loop Fusion Combine 2 or more loops into a single loop This cannot violate any dependencies between the loop bodies Several conditions which must be met for fusion to occur Often these conditions are not initially satisfied

4. March 14, 20024 Advantages of Loop Fusion Save increment and branch instructions Creates opportunities for data reuse Provide more instructions to instruction scheduler to balance the use of functional units

5. March 14, 20025 Disadvantages of Loop Fusion Increase code size effecting instruction cache performance Increase register pressure within a loop Could cause the formation of loops with more complex control flow

6. March 14, 20026 Background There has been extensive work done on loop fusion Most has focused on weighted loop fusion (Gao et al., Kennedy and McKinley, Megiddo and Sarkar) Extensive work has also been done it performing loop fusion to increase parallelism

7. March 14, 20027 Weighted Loop Fusion Associates non-negative weights with each pair of loop nests Weights are a measurement of the expected gain if the two loops are fused Gains include potential for array contraction, data reuse and improved local register allocation

8. March 14, 20028 Optimal Loop Fusion Fuse loops to optimize data reuse, taking into consideration resource constraints and register usage This problem is NP-Hard

9. March 14, 20029 Maximal Loop Fusion Our approach is to perform maximal loop fusion Fuse as many loops as possible, without considering resource constraints Fuse loops as soon as possible, not considering the consequences

10. March 14, 2002Allen & Kennedy, p. 150, 35310 Dominators and Post Dominators A node x in a directed graph G with a single exit node dominates node y in G if any path from the entry node of G to y must pass through x A node x in a directed graph G with a single exit node post-dominates node y in G if any path from y to the exit node of G must pass through x

11. March 14, 200211 Requirements for Loop Fusion i.Loops must have identical iteration counts (be conforming) ii.Loops must be control-flow equivalent iii.Loops must be adjacent iv.There cannot be any negative distance dependencies between the loops

12. March 14, 200212 Non-conforming Loops If iteration counts are different, one loop must be manipulated to make the iteration counts the same 1.Loop peeling 2.Introduce a guard into one of the loops

13. March 14, 200213 Loop Peeling Find the difference between the iteration count of the two loops (n) Duplicate the body of the loop with the higher iteration count n times Update the iteration count of the peeled loop

14. March 14, 200214 Loop Peeling Example while (i < 10) { a[i] = a[i - 1] * 2; i++; } while (j < 12) { b[j] = b[j - 1] - 2; j++; } while (i < 10) { a[i] = a[i - 1] * 2; i++; } while (j < 10) { b[j] = b[j - 1] - 2; j++; } b[j] = b[j - 1] - 2; j++; b[j] = b[j - 1] - 2; j++;

15. March 14, 200215 Guarding Iterations Increase the iteration count of the loop with fewer iterations Insert a guard branch around statements that would not normally be executed

16. March 14, 200216 Guarding Iterations Example while (i < 10) { a[i] = a[i - 1] * 2; i++; } while (j < 12) { b[j] = b[j - 1] - 2; j++; } while (i < 12) { if (i < 10) { a[i] = a[i - 1] * 2; i++; } while (j < 12) { b[j] = b[j - 1] - 2; j++; }

17. March 14, 200217 Loop Peeling Advantage: Does not generate control flow within a loop body Disadvantage: Generates additional code outside of loops, which could possible intervene with other loops

18. March 14, 200218 Guarding Iterations Advantages: Does not introduce intervening code Can be “undone” later Disadvantage: Generates control flow within a loop

19. March 14, 200219 Control Flow Equivalence Two loops are control-flow equivalent if when one executes, the other also executes Loop 1 BB Loop2 Loop 1 Loop 3 BB Loop2

20. March 14, 200220 Determining Control Flow Equivalence Use the concepts of dominators and post dominators. Two loops L1 and L2 are control-flow equivalent if the following two conditions are true: L1 dominates L2; and L2 post dominates L1.

21. March 14, 200221 Intervening Code Two loops are adjacent if there are no statements between the two loops Can be determined using the CFG: If the immediate successor of the first loop is the second loop, the two loops are adjacent If two loops are not adjacent, there is intervening code between them.

22. March 14, 200222 Dealing with Non-Adjacent Loops If two loops are not adjacent, we attempt to make them adjacent by moving the intervening code Intervening code can be moved: Above the first loop Below the second loop Both as long as no data dependencies are violated

23. March 14, 200223 Intervening Code Example Assume CFG has 20 nodes 0-5 are above Loop 1 17-19 are below Loop 2 What algorithm should be used to determine which nodes are between Loop1 and Loop2? Loop 1 Loop 2 6 7 89 101112 13 14 15 16

24. March 14, 200224 Gathering Intervening Code Given two loops L1 and L2, a basic block B is intervening code between L1 and L2 if and only if: oB is strictly dominated by L1 oB is not dominated by L2 Once the dominance relations are known, the set subtraction can be efficiently computed using bit vectors

25. March 14, 200225 Intervening Code Example Loop 1 Loop 2 6 7 89 101112 13 14 15 16 Loop 1 0000 0011 1111 1111 1111 1 Loop 2 0000 0000 0000 0000 1111 1 Difference 0000 0011 1111 1111 0000 0

26. March 14, 200226 Analyze Intervening Code Build a DDG of the intervening code Put all nodes with no predecessors into queue For each node in the queue: If there are no dependencies between the node and the loop Mark node as moveable Add all of the nodes immediate successors to the queue All nodes marked can be moved around the loop

27. March 14, 200227 Non-Adjacent loops example while (i < N) { a += i; i++; } b := a * 2; c := b + 6; g := 0; h := g + 10; if (c < 100) d := c/2; else e := c * 2; while (j < N) { f := g + 6; j++; } b := a * 2; c := b + 6; g := 0; if (c < 100) d := c/2; else e := c * 2; h := g + 10;

28. March 14, 200228 Non-Adjacent loops example while (i < N) { a += i; i++; } b := a * 2; c := b + 6; g := 0; h := g + 10; if (c < 100) d := c/2; else e := c * 2; while (j < N) { f := g + 6; j++; } g := 0; h := g + 10; while (i < N) { a += i; i++; } while (j < N) { f := g + 6; j++; } b := a * 2; c := b + 6; if (c < 100) d := c/2; else e := c * 2;

29. March 14, 200229 Non-Adjacent loops example b := a * 2; c := b + 6; g := 0; if (c < 100) d := c/2; else e := c * 2; h := g + 10; Node Queue b := a * 2; g := 0; DDGLoop 2 Moveable Nodes c := b + 6; if (c < 100) d := c/2; else e := c * 2; b := a * 2; c := b + 6; if (c < 100) d := c/2; else e := c * 2; while (j < N) { f := g + 6; j++; }

30. March 14, 200230 Non-Adjacent loops example b := a * 2; c := b + 6; g := 0; if (c < 100) d := c/2; else e := c * 2; h := g + 10; Node Queue b := a * 2; g := 0; DDGLoop 1 Moveable Nodes h := g + 10; g := 0; h := g + 10; while (i < N) { a += i; i++; }

31. March 14, 200231 Dependencies Preventing Fusion i = j = 1; while (i < 10) { a[i] = c[i] + 10; i++; } while (j < 10) { b[j] = a[j+1] * 2; j++; } Can the following loops be fused?

32. March 14, 200232 Dependencies Preventing Fusion If we look at the array access patterns of a[], we see the following a[i] = c[i] + 10; b[j] = a[j+1] * 2;

33. March 14, 200233 Dependencies Preventing Fusion By aligning the array access patterns, we get the following: a[i] = c[i] + 10; b[j] = a[j+1] * 2;

34. March 14, 200234 Loop Alignment i = j = 1; while (i < 10) { a[i] = c[i] + 10; i++; } while (j < 10) { b[j] = a[j+1] * 2; j++; } j = 1; i = 2 a[1] = c[1] + 10; while (i < 10) { a[i] = c[i] + 10; i++; } while (j < 10) { b[j] = a[j+1] * 2; j++; }

35. March 14, 200235 Loop Alignment Loop alignment can be used to remove dependencies between loop bodies Easy to do when all dependencies have the same distance Gets tricky when there are multiple dependencies with different distances

36. March 14, 200236 Putting it all together We’ve seen ways to deal with each of the preconditions of loop fusion If the conditions are not met, we apply transformations to try and modify the code If the transformations are successful, loop fusion can occur But in what order should these transformations be applied?

37. March 14, 200237 Loop Fusion Algorithm For each N i from outermost to innermost: Gather control equivalent loops in N i into LoopSets For each set S i in LoopSets remove non-eligible loops from S i FusedLoops = true Direction = forward while FusedLoops == true if |S i | < 2 break Compute Dominance Relation FusedLoops = LoopFusionPass(S i, Direction) Reverse Direction

38. March 14, 200238 Loop Fusion Algorithm LoopFusionPass(S, Direction) FusedLoops = false For each pair of loops L j and L k in S such that L j dominates L k in Direction if (DependenceDistance(L j, L k ) < 0) continue if (InterveningCode(L j, L k ) == true and IsInterveningCodeMoveable(L j, L k ) == false) continue d = | IterationCount(L j ) – IterationCount(L k ) | if (L j and L k are non-conforming and (d cannot be determined at compile time or d > MAXPEEL)) continue if (L j and L k are non-conforming) Peel iterations MoveInterveningCode(L j, L k ) if InterveningCode(L j, L k ) == false FuseLoops(L j, L k ) FusedLoops = true Return FusedLoops

39. March 14, 200239 Example L1: do i1 = 1, n a(i1) = a(i1) * k1 end do L2: do i2 = 1, n-1 d(i2) = a(i2) - b(i2+1) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do Loop Set L1 L2 L3 L4

40. March 14, 200240 Peeling Loop 1 L1: do i1 = 1, n a(i1) = a(i1) * k1 end do L2: do i2 = 1, n-1 d(i2) = a(i2) - b(i2+1) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do S7: a(1) = a(1) * k1 L1: do i1 = 1, n-1 a(i1+1) = a(i1+1) * k1 end do L2: do i2 = 1, n-1 d(i2) = a(i2) - b(i2+1) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do

41. March 14, 200241 Fuse L1 and L2 S7: a(1) = a(1) * k1 L5: do i5 = 1, n-1 a(i5+1) = a(i5+1) * k1 d(i5) = a(i5) - b(i5+1) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do S7: a(1) = a(1) * k1 L1: do i1 = 1, n-1 a(i1+1) = a(i1+1) * k1 end do L2: do i2 = 1, n-1 d(i2) = a(i2) - b(i2+1) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do

42. March 14, 200242 Compare L5 and L3 We now compare loops L5 and L3 They are not adjacent, but the intervening code can move Difference in iteration count is not know, so fusion fails S7: a(1) = a(1) * k1 L5: do i5 = 1, n-1 a(i5+1) = a(i5+1) * k1 d(i5) = a(i5) - b(i5+1) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do

43. March 14, 200243 Compare L5 and L4 Intervening Code S7: a(1) = a(1) * k1 L5: do i5 = 1, n-1 a(i5+1) = a(i5+1) * k1 d(i5) = a(i5) - b(i5+1) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m

44. March 14, 200244 Peel L5 S7: a(1) = a(1) * k1 L5: do i5 = 1, n-1 a(i5+1) = a(i5+1) * k1 d(i5) = a(i5) - b(i5+1) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 L5: do i5 = 1, n-2 a(i5+2) = a(i5+2) * k1 d(i5+1) = a(i5+1) - b(i5+2) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do

45. March 14, 200245 Move Intervening Code S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 S1: ds = 0.0 S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L5: do i5 = 1, n-2 a(i5+2) = a(i5+2) * k1 d(i5+1) = a(i5+1) - b(i5+2) * k2 end do L3: do i3 = 1, m ds = ds + d(i3) end do L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 L5: do i5 = 1, n-2 a(i5+2) = a(i5+2) * k1 d(i5+1) = a(i5+1) - b(i5+2) * k2 end do S1: ds = 0.0 L3: do i3 = 1, m ds = ds + d(i3) end do S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do

46. March 14, 200246 Reverse Pass S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 S1: ds = 0.0 S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L5: do i5 = 1, n-2 a(i5+2) = a(i5+2) * k1 d(i5+1) = a(i5+1) - b(i5+2) * k2 end do L3: do i3 = 1, m ds = ds + d(i3) end do L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do Loop Set L1 L3 L4 Sorted in Reverse Dominance Direction L1 L3 L4

47. March 14, 200247 Compare L4 and L3 No dependencies to prevent fusion Iteration count cannot be determined at compile time Fusion fails S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 S1: ds = 0.0 S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L5: do i5 = 1, n-2 a(i5+2) = a(i5+2) * k1 d(i5+1) = a(i5+1) - b(i5+2) * k2 end do L3: do i3 = 1, m ds = ds + d(i3) end do L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do

48. March 14, 200248 Compare L4 and L5 Intervening Code L3: do i3 = 1, m ds = ds + d(i3) end do S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 S1: ds = 0.0 S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L5: do i5 = 1, n-2 a(i5+2) = a(i5+2) * k1 d(i5+1) = a(i5+1) - b(i5+2) * k2 end do L3: do i3 = 1, m ds = ds + d(i3) end do L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do

49. March 14, 200249 Move Intervening Code S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 S1: ds = 0.0 S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L5: do i5 = 1, n-2 a(i5+2) = a(i5+2) * k1 d(i5+1) = a(i5+1) - b(i5+2) * k2 end do L3: do i3 = 1, m ds = ds + d(i3) end do L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 S1: ds = 0.0 S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L5: do i5 = 1, n-2 a(i5+2) = a(i5+2) * k1 d(i5+1) = a(i5+1) - b(i5+2) * k2 end do L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do L3: do i3 = 1, m ds = ds + d(i3) end do

50. March 14, 200250 Fuse L4 and L1 S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 S1: ds = 0.0 S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L6: do i5 = 1, n-2 a(i6+2) = a(i6+2) * k1 d(i6+1) = a(i6+1) - b(i6+2) * k2 b(i6) = a(i6) + b(i6) / c(i6) end do L3: do i3 = 1, m ds = ds + d(i3) end do S7: a(1) = a(1) * k1 S8: a(2) = a(2) * k1 S9: d(1) = a(1) - b(2) * k2 S1: ds = 0.0 S2: if (n<m) S3: c(n-2) = n S4: else S5: c(n-2) = m L5: do i5 = 1, n-2 a(i5+2) = a(i5+2) * k1 d(i5+1) = a(i5+1) - b(i5+2) * k2 end do L4: do i4 = 1, n-2 b(i4) = a(i4) + b(i4) / c(i4) end do L3: do i3 = 1, m ds = ds + d(i3) end do