1. Chapter 2 Instruction-Level Parallelism and Its Exploitation

2. See Subset of the MIPS64 Instructions in the back cover of the textbook.
CSCE 614 Fall 2009

3. Instruction-Level Parallelism (ILP)
Instructions are evaluated in parallel.
Pipelining
Two approaches to exploiting ILP
Dynamic & Hardware-dependent (chapter 2)
Intel Pentium Series, Athlon, MIPS R10000/12000, Sun UltraSPARC III, PowerPC, …
Static & Software-dependent (Appendix A, Appendix G)
IA-64, Intel Itanium, embedded processors
CSCE 614 Fall 2009

4. Ideal CPI + Structural stalls + Data hazard stalls + Control stalls
Pipeline CPI =
Ideal CPI + Structural stalls
+ Data hazard stalls + Control stalls
CSCE 614 Fall 2009

5. Techniques to Decrease Pipeline CPI (p. 67)
Forwarding and Bypassing
Delayed Branches and Simple Branch Scheduling
Basic Dynamic Scheduling (Scoreboarding)
Dynamic Scheduling with Renaming
Branch Prediction
Issuing Multiple Instructions per Cycle
Hardware Speculation
Dynamic Memory Disambiguation
CSCE 614 Fall 2009

6. Techniques to Decrease Pipeline CPI (p. 67)
Loop Unrolling
Basic Compiler Pipeline Scheduling
Compiler Dependence Analysis, Software Pipelining, Trace Scheduling
Hardware Support for Compiler Speculation
CSCE 614 Fall 2009

7. Data Dependences
If two instructions are parallel, they can execute simultaneously.
If two instructions are dependent, they must be executed in order.
How to determine an instruction is dependent on anther instruction?
CSCE 614 Fall 2009

8. Data Dependences Data dependences (True data dependences)
Name dependences
Control dependences
An instruction j is data dependent on instruction i if either
i produces a result that may be used by j, or
j is data dependent on instruction k, and k is data dependent on i.
CSCE 614 Fall 2009

9. Loop: L.D F0, 0(R1) ;F0=array element
ADD.D F4, F0, F2 ;add scalar in F2
S.D F4, 0(R1) ;store the result
DADDIU R1, R1, #-8 ;decrement pointer 8 bytes
BNE R1, R2, LOOP ;branch R1 != R2
Floating-point data dependences
Integer data dependence
CSCE 614 Fall 2009

10. Data Dependences The order must be preserved for correct execution.
If two instructions are data dependent, they cannot execute simultaneously or be completely overlapped.
Data dependence between DADDIU and BNE => Branch test for the MIPS pipeline in the ID stage (2nd stage).
CSCE 614 Fall 2009

11. Pipelined Datapath
CSCE 614 Fall 2009

12. Data Dependences A dependence indicates the possibility of a hazard,
determines the order in which results must be calculated, and
sets an upper bound on how much parallelism can be possibly be exploited.
CSCE 614 Fall 2009

13. How to Overcome a Dependence
Maintaining the dependence but avoiding a hazard
Code scheduling (by the compiler or by the hardware)
Eliminating a dependence by transforming the code
CSCE 614 Fall 2009

14. Name Dependence
Occurs when two instructions use the same register or memory location (name), but there is no flow of data between the instructions associated with that name.
When i precedes j in program order:
Antidependence: Instruction j writes a register or memory location that instruction i reads.
Output dependence: Instructions i and j write the same register or memory location.
No value transmitted between instructions.
CSCE 614 Fall 2009

15. Register Renaming
Instructions involved in a name dependence can execute simultaneously or be reordered, if the name (register number or memory location) used in the instructions is changed so the instructions do not conflict. (Especially for register operands)
Statically by a compiler or dynamically by the hardware.
CSCE 614 Fall 2009

16. Hazards
A hazard is created whenever there is a dependence between instructions, and they are close enough that the overlap during execution, caused by pipelining, or other reordering of instructions, would change the order of access to the operand involved in the dependence.
CSCE 614 Fall 2009

17. 3 Data Hazards
The goal of S/W and H/W Techniques in our course is to preserve the program order only where it affects the outcome of the program to maximize ILP.
When instruction i occurs before instruction j in program order,
RAW (Read after Write): j tries to read a source before i writes it.
WAW (Write after Write): j tries to write an operand before it is written by i.
WAR (Write after Read): j tries to write a destination before it is read by i.
CSCE 614 Fall 2009

18. Control Dependences Caused by branch instructions
An instruction that is control dependent on a branch cannot be moved before the branch.
An instruction that is not control dependent on a branch cannot be moved after the branch.
CSCE 614 Fall 2009

19. Control dependence is not the critical property that must be preserved.
We can violate the control dependences, if we can do so without affecting the correctness of the program. (e.g. branch prediction)
CSCE 614 Fall 2009

20. Basic Compiler Techniques for Exposing ILP

21. Goal: to keep a pipeline full.
To avoid a pipeline stall, a dependent instruction must be separated from the source instruction by a distance in clock cycles equal to the pipeline latency of that source instruction.
Goal: to keep a pipeline full.
CSCE 614 Fall 2009

22. Basic Pipeline Scheduling and Loop Unrolling

23. Latencies Inst. producing result Inst. using result Latency in cycles
FP ALU op
Another FP op 3
Store double 2
Load double 1
Latency: the number of clock cycles needed to avoid a stall
between a producer and a consumer
Branch: 1, Integer ALU op – branch: 1
Integer load: 1 Integer ALU - integer ALU: 0
CSCE 614 Fall 2009
Functional units are fully pipelined or replicated. => No structural hazard

24. Example for ( i = 1000; i > 0; i = i – 1) x[i] = x[i] + s;
MIPS Assembly code
Loop: L.D F0, 0(R1)
ADD.D F4, F0, F2
S.D F4, 0(R1)
DADDIU R1, R1, # -8
BNE R1, R2, LOOP
CSCE 614 Fall 2009

25. Without Any Scheduling
Clock cycle issued
Loop: L.D F0, 0(R1) 1
stall 2
ADD.D F4, F0, F2 3
stall 4
stall 5
S.D F4, 0(R1) 6
DADDIU R1, R1, #
stall 8
BNE R1, R2, LOOP 9
stall 10
CSCE 614 Fall 2009

26. With Scheduling Clock cycle issued Loop: L.D F0, 0(R1) 1
DADDIU R1, R1, #
ADD.D F4, F0, F2 3
stall 4
BNE R1, R2, LOOP 5
S.D F4, 8(R1) 6
delayed branch
not trivial
CSCE 614 Fall 2009

27. The actual work of operating on the array element takes 3 cycles (load, add, store).
The remaining 3 cycles
Loop overhead (DADDIU, BNE)
Stall
To eliminate the 3 cycles, we need to get more operations within the loop relative to the number of overhead instructions.
=> Loop Unrolling
CSCE 614 Fall 2009

28. Reducing Loop Overhead
Loop Unrolling
Simple scheme for increasing the number of instructions relative to the branch and overhead instructions
Simply replicates the loop body multiple times, adjusting the loop termination code.
Improves scheduling
It allows instructions from different iterations to be scheduled together.
Uses different registers for each iteration.
CSCE 614 Fall 2009

29. Unrolled Loop (No Scheduling)
Clock cycle issued
Loop: L.D F0, 0(R1) 1 2
ADD.D F4, F0, F
S.D F4, 0(R1) 6
L.D F6, -8(R1) 7 8
ADD.D F8, F6, F
S.D F8, -8(R1) 12
L.D F10, -16(R1)
ADD.D F12, F10, F
S.D F12, -16(R1) 18
L.D F14, -24(R1)
ADD.D F16, F14, F
S.D F16, -24(R1) 24
DADDIU R1, R1, #
BNE R1, R2, LOOP
DADDIU
and
BNE
dropped
CSCE 614 Fall 2009

30. Loop Unrolling
Loop unrolling is normally done early in the compilation process, so that redundant computations can be exposed and eliminated by the optimizer.
Unrolling improves the performance of the loop by eliminating overhead instructions.
CSCE 614 Fall 2009

31. Loop Unrolling (Scheduling)
Clock cycle issued
Loop: L.D F0, 0(R1) 1
L.D F6, -8(R1) 2
L.D F10, -16(R1) 3
L.D F14, -24(R1) 4
ADD.D F4, F0, F2 5
ADD.D F8, F6, F2 6
ADD.D F12, F10, F2 7
ADD.D F16, F14, F2 8
S.D F4, 0(R1) 9
S.D F8, -8(R1) 10
DADDIU R1, R1, #
S.D F12, 16(R1) 12
BNE R1, R2, LOOP 13
S.D F16, 8(R1) 14
CSCE 614 Fall 2009

32. Summary
The key to most hardware and software ILP techniques is to know when and how the ordering among instructions may be changed.
This process must be performed in a methodical fashion either by a compiler or by hardware.
CSCE 614 Fall 2009

33. To obtain the final unrolled code,
Determine that it is legal to move the S.D after the DADDIU and BNE, and find the amount to adjust the S.D offset.
Determine that unrolling the loop will be useful by finding that the loop iterations are independent, except for the loop maintenance code.
Use different registers to avoid unnecessary constraints.
Eliminate the extra test and branch instructions and adjust the loop termination and iteration code.
CSCE 614 Fall 2009

34. Determine that the loads and stores in the unrolled loop can be interchanged by observing that the loads and stores from different iterations are independent. This transformation requires analyzing the memory addresses and finding that they do not refer to the same address.
Schedule the code, preserving any dependences needed to yield the same result as the original code.
CSCE 614 Fall 2009

35. Loop Unrolling I (Unoptimized, No Delayed Branch)
Loop: L.D F0, 0(R1)
ADD.D F4, F0, F2
S.D F4, 0(R1)
DADDIU R1, R1, #-8
L.D F0, 0(R1)
DADDIU R1, R1, # -8
BNE R1, R2, LOOP
By symbolically
computing the
intermediate value
of R1
CSCE 614 Fall 2009

36. Loop Unrolling I (Unoptimized, No Delayed Branch)
Loop: L.D F0, 0(R1)
ADD.D F4, F0, F2
S.D F4, 0(R1)
L.D F0, -8(R1)
S.D F4, -8(R1)
L.D F0, -16(R1)
S.D F4, -16(R1)
L.D F0, -24(R1)
S.D F4, -24(R1)
DADDIU R1, R1, # -32
BNE R1, R2, LOOP
Remove name dependences
using Register Renaming
name dependence
true dependence
CSCE 614 Fall 2009

37. Loop Unrolling II (Register Renaming)
Loop: L.D F0, 0(R1)
ADD.D F4, F0, F2
S.D F4, 0(R1)
L.D F6, -8(R1)
ADD.D F8, F6, F2
S.D F8, -8(R1)
L.D F10, -16(R1)
ADD.D F12, F10, F2
S.D F12, -16(R1)
L.D F14, -24(R1)
ADD.D F16, F14, F2
S.D F16, -24(R1)
DADDIU R1, R1, # -32
BNE R1, R2, LOOP
true dependence
CSCE 614 Fall 2009

38. With the renaming, the copies of each loop body become independent and can be overlapped or executed in parallel.
Problem: potential shortfall in registers
Register pressure
It arises because scheduling code to increase ILP causes the number of live values to increase. It may not be possible to allocate all the live values to registers.
The combination of unrolling and aggressive scheduling can cause this problem.
CSCE 614 Fall 2009

39. Loop unrolling is a simple but useful method for increasing the size of straight-line code fragments that can be scheduled effectively.
CSCE 614 Fall 2009