Chapter 2 Instruction-Level Parallelism and Its Exploitation

See Subset of the MIPS64 Instructions in the back cover of the textbook. CSCE 614 Fall 2009

Instruction-Level Parallelism (ILP) Instructions are evaluated in parallel. Pipelining Two approaches to exploiting ILP Dynamic & Hardware-dependent (chapter 2) Intel Pentium Series, Athlon, MIPS R10000/12000, Sun UltraSPARC III, PowerPC, … Static & Software-dependent (Appendix A, Appendix G) IA-64, Intel Itanium, embedded processors CSCE 614 Fall 2009

Ideal CPI + Structural stalls + Data hazard stalls + Control stalls Pipeline CPI = Ideal CPI + Structural stalls + Data hazard stalls + Control stalls CSCE 614 Fall 2009

Techniques to Decrease Pipeline CPI (p. 67) Forwarding and Bypassing Delayed Branches and Simple Branch Scheduling Basic Dynamic Scheduling (Scoreboarding) Dynamic Scheduling with Renaming Branch Prediction Issuing Multiple Instructions per Cycle Hardware Speculation Dynamic Memory Disambiguation CSCE 614 Fall 2009

Techniques to Decrease Pipeline CPI (p. 67) Loop Unrolling Basic Compiler Pipeline Scheduling Compiler Dependence Analysis, Software Pipelining, Trace Scheduling Hardware Support for Compiler Speculation CSCE 614 Fall 2009

Data Dependences If two instructions are parallel, they can execute simultaneously. If two instructions are dependent, they must be executed in order. How to determine an instruction is dependent on anther instruction? CSCE 614 Fall 2009

Data Dependences Data dependences (True data dependences) Name dependences Control dependences An instruction j is data dependent on instruction i if either i produces a result that may be used by j, or j is data dependent on instruction k, and k is data dependent on i. CSCE 614 Fall 2009

Loop: L.D F0, 0(R1) ;F0=array element ADD.D F4, F0, F2 ;add scalar in F2 S.D F4, 0(R1) ;store the result DADDIU R1, R1, #-8 ;decrement pointer 8 bytes BNE R1, R2, LOOP ;branch R1 != R2 Floating-point data dependences Integer data dependence CSCE 614 Fall 2009

Data Dependences The order must be preserved for correct execution. If two instructions are data dependent, they cannot execute simultaneously or be completely overlapped. Data dependence between DADDIU and BNE => Branch test for the MIPS pipeline in the ID stage (2nd stage). CSCE 614 Fall 2009

Pipelined Datapath CSCE 614 Fall 2009

Data Dependences A dependence indicates the possibility of a hazard, determines the order in which results must be calculated, and sets an upper bound on how much parallelism can be possibly be exploited. CSCE 614 Fall 2009

How to Overcome a Dependence Maintaining the dependence but avoiding a hazard Code scheduling (by the compiler or by the hardware) Eliminating a dependence by transforming the code CSCE 614 Fall 2009

Name Dependence Occurs when two instructions use the same register or memory location (name), but there is no flow of data between the instructions associated with that name. When i precedes j in program order: Antidependence: Instruction j writes a register or memory location that instruction i reads. Output dependence: Instructions i and j write the same register or memory location. No value transmitted between instructions. CSCE 614 Fall 2009

Register Renaming Instructions involved in a name dependence can execute simultaneously or be reordered, if the name (register number or memory location) used in the instructions is changed so the instructions do not conflict. (Especially for register operands) Statically by a compiler or dynamically by the hardware. CSCE 614 Fall 2009

Hazards A hazard is created whenever there is a dependence between instructions, and they are close enough that the overlap during execution, caused by pipelining, or other reordering of instructions, would change the order of access to the operand involved in the dependence. CSCE 614 Fall 2009

3 Data Hazards The goal of S/W and H/W Techniques in our course is to preserve the program order only where it affects the outcome of the program to maximize ILP. When instruction i occurs before instruction j in program order, RAW (Read after Write): j tries to read a source before i writes it. WAW (Write after Write): j tries to write an operand before it is written by i. WAR (Write after Read): j tries to write a destination before it is read by i. CSCE 614 Fall 2009

Control Dependences Caused by branch instructions An instruction that is control dependent on a branch cannot be moved before the branch. An instruction that is not control dependent on a branch cannot be moved after the branch. CSCE 614 Fall 2009

Control dependence is not the critical property that must be preserved. We can violate the control dependences, if we can do so without affecting the correctness of the program. (e.g. branch prediction) CSCE 614 Fall 2009

Basic Compiler Techniques for Exposing ILP

Goal: to keep a pipeline full. To avoid a pipeline stall, a dependent instruction must be separated from the source instruction by a distance in clock cycles equal to the pipeline latency of that source instruction. Goal: to keep a pipeline full. CSCE 614 Fall 2009

Basic Pipeline Scheduling and Loop Unrolling

Latencies Inst. producing result Inst. using result Latency in cycles FP ALU op Another FP op 3 Store double 2 Load double 1 Latency: the number of clock cycles needed to avoid a stall between a producer and a consumer Branch: 1, Integer ALU op – branch: 1 Integer load: 1 Integer ALU - integer ALU: 0 CSCE 614 Fall 2009 Functional units are fully pipelined or replicated. => No structural hazard

Example for ( i = 1000; i > 0; i = i – 1) x[i] = x[i] + s; MIPS Assembly code Loop: L.D F0, 0(R1) ADD.D F4, F0, F2 S.D F4, 0(R1) DADDIU R1, R1, # -8 BNE R1, R2, LOOP CSCE 614 Fall 2009

Without Any Scheduling Clock cycle issued Loop: L.D F0, 0(R1) 1 stall 2 ADD.D F4, F0, F2 3 stall 4 stall 5 S.D F4, 0(R1) 6 DADDIU R1, R1, # -8 7 stall 8 BNE R1, R2, LOOP 9 stall 10 CSCE 614 Fall 2009

With Scheduling Clock cycle issued Loop: L.D F0, 0(R1) 1 DADDIU R1, R1, # -8 2 ADD.D F4, F0, F2 3 stall 4 BNE R1, R2, LOOP 5 S.D F4, 8(R1) 6 delayed branch not trivial CSCE 614 Fall 2009

The actual work of operating on the array element takes 3 cycles (load, add, store). The remaining 3 cycles Loop overhead (DADDIU, BNE) Stall To eliminate the 3 cycles, we need to get more operations within the loop relative to the number of overhead instructions. => Loop Unrolling CSCE 614 Fall 2009

Reducing Loop Overhead Loop Unrolling Simple scheme for increasing the number of instructions relative to the branch and overhead instructions Simply replicates the loop body multiple times, adjusting the loop termination code. Improves scheduling It allows instructions from different iterations to be scheduled together. Uses different registers for each iteration. CSCE 614 Fall 2009

Unrolled Loop (No Scheduling) Clock cycle issued Loop: L.D F0, 0(R1) 1 2 ADD.D F4, F0, F2 3 4 5 S.D F4, 0(R1) 6 L.D F6, -8(R1) 7 8 ADD.D F8, F6, F2 9 10 11 S.D F8, -8(R1) 12 L.D F10, -16(R1) 13 14 ADD.D F12, F10, F2 15 16 17 S.D F12, -16(R1) 18 L.D F14, -24(R1) 19 20 ADD.D F16, F14, F2 21 22 23 S.D F16, -24(R1) 24 DADDIU R1, R1, # -32 25 26 BNE R1, R2, LOOP 27 28 DADDIU and BNE dropped CSCE 614 Fall 2009

Loop Unrolling Loop unrolling is normally done early in the compilation process, so that redundant computations can be exposed and eliminated by the optimizer. Unrolling improves the performance of the loop by eliminating overhead instructions. CSCE 614 Fall 2009

Loop Unrolling (Scheduling) Clock cycle issued Loop: L.D F0, 0(R1) 1 L.D F6, -8(R1) 2 L.D F10, -16(R1) 3 L.D F14, -24(R1) 4 ADD.D F4, F0, F2 5 ADD.D F8, F6, F2 6 ADD.D F12, F10, F2 7 ADD.D F16, F14, F2 8 S.D F4, 0(R1) 9 S.D F8, -8(R1) 10 DADDIU R1, R1, # -32 11 S.D F12, 16(R1) 12 BNE R1, R2, LOOP 13 S.D F16, 8(R1) 14 CSCE 614 Fall 2009

Summary The key to most hardware and software ILP techniques is to know when and how the ordering among instructions may be changed. This process must be performed in a methodical fashion either by a compiler or by hardware. CSCE 614 Fall 2009

To obtain the final unrolled code, Determine that it is legal to move the S.D after the DADDIU and BNE, and find the amount to adjust the S.D offset. Determine that unrolling the loop will be useful by finding that the loop iterations are independent, except for the loop maintenance code. Use different registers to avoid unnecessary constraints. Eliminate the extra test and branch instructions and adjust the loop termination and iteration code. CSCE 614 Fall 2009

Determine that the loads and stores in the unrolled loop can be interchanged by observing that the loads and stores from different iterations are independent. This transformation requires analyzing the memory addresses and finding that they do not refer to the same address. Schedule the code, preserving any dependences needed to yield the same result as the original code. CSCE 614 Fall 2009

Loop Unrolling I (Unoptimized, No Delayed Branch) Loop: L.D F0, 0(R1) ADD.D F4, F0, F2 S.D F4, 0(R1) DADDIU R1, R1, #-8 L.D F0, 0(R1) DADDIU R1, R1, # -8 BNE R1, R2, LOOP By symbolically computing the intermediate value of R1 CSCE 614 Fall 2009

Loop Unrolling I (Unoptimized, No Delayed Branch) Loop: L.D F0, 0(R1) ADD.D F4, F0, F2 S.D F4, 0(R1) L.D F0, -8(R1) S.D F4, -8(R1) L.D F0, -16(R1) S.D F4, -16(R1) L.D F0, -24(R1) S.D F4, -24(R1) DADDIU R1, R1, # -32 BNE R1, R2, LOOP Remove name dependences using Register Renaming name dependence true dependence CSCE 614 Fall 2009

Loop Unrolling II (Register Renaming) Loop: L.D F0, 0(R1) ADD.D F4, F0, F2 S.D F4, 0(R1) L.D F6, -8(R1) ADD.D F8, F6, F2 S.D F8, -8(R1) L.D F10, -16(R1) ADD.D F12, F10, F2 S.D F12, -16(R1) L.D F14, -24(R1) ADD.D F16, F14, F2 S.D F16, -24(R1) DADDIU R1, R1, # -32 BNE R1, R2, LOOP true dependence CSCE 614 Fall 2009

With the renaming, the copies of each loop body become independent and can be overlapped or executed in parallel. Problem: potential shortfall in registers Register pressure It arises because scheduling code to increase ILP causes the number of live values to increase. It may not be possible to allocate all the live values to registers. The combination of unrolling and aggressive scheduling can cause this problem. CSCE 614 Fall 2009

Loop unrolling is a simple but useful method for increasing the size of straight-line code fragments that can be scheduled effectively. CSCE 614 Fall 2009