1. HazardsCS510 Computer Architectures Lecture 7 - 1 Lecture 7 Pipeline Hazards

2. HazardsCS510 Computer Architectures Lecture 7 - 2 Pipelining Lessons A B C D 6 PM 789 TaskOrderTaskOrder Time 30 40 20 Pipelining doesn’t help latency of single task, it helps throughput of entire workload Pipeline rate limited by slowest pipeline stage Multiple tasks operating simultaneously Potential speedup = Number pipe stagesPotential speedup = Number pipe stages Unbalanced lengths of pipe stages reduces speedup Time to “fill” pipeline and time to “drain” it reduces speedup

3. HazardsCS510 Computer Architectures Lecture 7 - 3 Its Not That Easy to Achieve the Promised Performance Limits to pipelining: Hazards prevent the next instruction from executing during its designated clock cycle –Structural hazards: HW cannot support this combination of instructions –Data hazards: Instruction depends on result of prior instruction still in the pipeline –Control hazards: Pipelining of branches and other instructions that change the PC Common solution is to stall the pipeline until the hazard is resolved, inserting one or more “bubbles”, i.e., idle clock cycles, in the pipeline

4. HazardsCS510 Computer Architectures Lecture 7 - 4 Structural Hazards /Memory Instruction Order LOAD Instr 1 Instr 2 Mem Reg ALU Mem Reg Mem Reg ALU Mem Reg Instr 3 Reg ALU Reg Mem Reg ALU Mem Time(clock cycles) CC1 CC2 CC3 CC4 CC5 CC6 CC7 CC8 CC9 Mem Reg ALU Mem Reg Instr 4 Mem Operation on Memory by 2 different instructions in the same clock cycle

5. HazardsCS510 Computer Architectures Lecture 7 - 5 Structural Hazards with Single-Port Memory Instruction Order LOAD Instr 1 Instr 2 Time(clock cycles) CC1 CC2 CC3 CC4 CC5 CC6 CC7 CC8 CC9 Mem Reg ALU Mem Reg Mem Reg ALU Mem Reg Instr 3 Stall Reg ALU Reg Mem Reg ALU Mem Stall Mem Reg ALU Instr 3 Mem 3 cycles stall with 1-port memory

6. HazardsCS510 Computer Architectures Lecture 7 - 6 Avoiding Structural Hazard with Dual-Port Memory Instruction Order Time(clock cycles) CC1 CC2 CC3 CC4 CC5 CC6 CC7 CC8 CC9 LOAD IM Reg ALU RegDM Instr 1 Instr 2 Instr 3 Instr 4 Instr 5 IM DM IM Reg ALU DMReg ALU DMReg IM Reg ALU DMReg IM Reg ALU DMReg IM Reg ALU DM IM DM IM DM IM DM IM DM IM DM No stall with 2-port memory

7. HazardsCS510 Computer Architectures Lecture 7 - 7 Speed Up Equation for Pipelining Ideal CPI for pipelined machines is almost always 1 Ideal CPI = CPI unpipelined /Pipeline depth(Number of pipeline stages) x Speedup = Ideal CPI x Pipeline depth Clock Cycle unpipelined CPI pipelined Clock Cycle pipelined x Speedup from pipelining Ave Instr Time unpipelined Ave Instr Time pipelined CPI unpipelined x Clock Cycle unpipelined CPI pipelined x Clock Cycle pipelined CPI unpipelined Clock Cycle unpipelined CPI pipelined Clock Cycle pipelined

8. HazardsCS510 Computer Architectures Lecture 7 - 8 Speed Up Equation for Pipelining CPI pipelined = Ideal CPI + Pipeline stall clock cycles per instr = 1 + Pipeline stall clock cycles per instr x Speedup = Ideal CPI x Pipeline depth Clock Cycle unpipelined Ideal CPI + Pipeline stall CPI Clock Cycle pipelined x Speedup = Pipeline depth Clock Cycle unpipelined 1 + Pipeline stall CPI Clock Cycle pipelined

9. HazardsCS510 Computer Architectures Lecture 7 - 9 Dual-Port vs Single-Port Memory Machine A: 2-port memory(needs no stall for Load); same clock cycle as unpipelined machine Machine B: 1-ported memory(needs 3 cycles stall for Load); 1.05 times faster clock rate than the unpipelined machine Ideal CPI = 1 for both Loads are 40% of instructions executed Machine A is 1.15 times faster SpeedUp A = [Pipeline Depth/(1 + 0)] x (clock unpipe /clock pipe ) = Pipeline Depth SpeedUp B = Pipeline Depth/(1 + 0.4 x 3) x (clock unpipe /(clock unpipe / 1.05) = (Pipeline Depth/1.2) x 1.05 = 0.87 x Pipeline Depth SpeedUp A / SpeedUp B = Pipeline Depth/(0.87 x Pipeline Depth) = 1.15

10. HazardsCS510 Computer Architectures Lecture 7 - 10 Data Hazard on Registers ADD R1,R2,R3 SUB R4,R1,R3 AND R6,R1,R7 OR R8,R1,R9 XOR R10,R11,R1 CC1 CC2 CC3 CC4 CC5 CC6 CC7 CC8 CC9 Mem Reg ALU Reg ALU MemReg Mem ALU MemReg ALU MemReg Mem ALU MemReg Mem Reg Time(clock cycles) R1 Re Reg Reg Reg Reg

11. HazardsCS510 Computer Architectures Lecture 7 - 11 Data Hazard on Registers Registers can be made to read and store in the same cycle such that data is stored in the first half of the clock cycle, and that data can be read in the second half of the same clock cycle Clcok Cycle Register Ri Store into Ri Read from Ri

12. HazardsCS510 Computer Architectures Lecture 7 - 12 Data Hazard on Registers ADD R1,R2,R3 SUB R4,R1,R3 AND R6,R1,R7 OR R8,R1,R9 XOR R10,R11,R1 CC1 CC2 CC3 CC4 CC5 CC6 CC7 CC8 CC9 Time(clock cycles) R1 Reg Needs to Stall 2 cycles Reg

13. HazardsCS510 Computer Architectures Lecture 7 - 13 Three Generic Data Hazards Instr i followed by Instr j Read After Write (RAW) Instr j tries to read operand before Instr i writes it Instr i LW R1, 0(R2) Instr j SUBR 4, R1, R5

14. HazardsCS510 Computer Architectures Lecture 7 - 14 Three Generic Data Hazards Instr I followed by Instr J Write After Read (WAR) Instr j tries to write operand before Instr i reads it Instr i ADD R1, R2, R3 Instr j LW R2, 0(R5) Can’t happen in DLX 5 stage pipeline because: – All instructions take 5 stages, – Reads are always in stage 2, and – Writes are always in stage 5

15. HazardsCS510 Computer Architectures Lecture 7 - 15 Three Generic Data Hazards Instr I followed by Instr J Write After Write (WAW) Instr j tries to write operand before Instr i writes it – Leaves wrong result ( Instr i not Instr j ) Instr i LW R1, 0(R2) Instr j LW R1, 0(R3) Can’t happen in DLX 5 stage pipeline because: – All instructions take 5 stages, and – Writes are always in stage 5 Will see WAR and WAW in later more complicated pipes

16. HazardsCS510 Computer Architectures Lecture 7 - 16 Forwarding to Avoid Data Hazards Time(clock cycles) ADD R1,R2,R3 SUB R4,R1,R3 AND R6,R1,R7 OR R8,R1,R9 XOR R10,R11,R1 Mem Reg ALU Reg ALU MemReg Mem Reg ALU MemReg ALU MemReg Mem Reg ALU MemReg CC1 CC2 CC3 CC4 CC5 CC6 CC7 CC8 CC9 Mem

17. HazardsCS510 Computer Architectures Lecture 7 - 17 HW Change for Forwarding MUX Zero? Data Memory ALU D/A Buffer A/M BufferM/W Buffer

18. HazardsCS510 Computer Architectures Lecture 7 - 18

19. HazardsCS510 Computer Architectures Lecture 7 - 19 Load Delay Due to Data Hazard LOAD R1,0(R2) Time(clock cycles) AND R6,R1,R7 IM Reg ALU DMReg OR R8,R1,R9 Reg ALU DM IM SUB R4,R1,R6 Reg ALU DMReg IM Load Delay =2cycles Reg ALU DMReg IM Reg ALU DMReg IM Reg ALU DMReg IM Reg ALU RegDM Reg ALU DMReg IM

20. HazardsCS510 Computer Architectures Lecture 7 - 20 Load Delay with Forwarding LOAD R1,0(R2) Time(clock cycles) IM Reg ALU RegDM SUB R4,R1,R6 AND R6,R1,R7 OR R8,R1,R9 IM Reg ALU DMReg ALU DMReg IM We need to add HW, called Pipeline Interlock IM Reg ALU DMReg ALU DMReg IM Reg ALU DMReg IM Load Delay with Forwarding=1cycle

21. HazardsCS510 Computer Architectures Lecture 7 - 21 Try to produce fast code for a = b + c; d = e - f; assuming a, b, c, d,e, and f are in memory. Software Scheduling to Avoid Load Hazards Fast code: LW Rb,b LW Rc,c LW Re,e ADD Ra,Rb,Rc LW Rf,f SW a,Ra SUB Rd,Re,Rf SWd,Rd Slow code(with forwarding): LW Rb,b LW Rc,c ADD Ra,Rb,Rc SW a,Ra LW Re,e LW Rf,f SUB Rd,Re,Rf SWd,Rd RAW Stall

22. HazardsCS510 Computer Architectures Lecture 7 - 22 % loads stalling pipeline 0%20%40%60%80% tex spice gcc 25% 14% 31% 65% 42% 54% scheduledunscheduled Compiler Avoiding Load Stalls

23. HazardsCS510 Computer Architectures Lecture 7 - 23 Mem Stage WB Stage IF Stage ID StageEX Stage Instr. Memory Sign Ext Zero? Data Memory PC MUX Add ALU Reg File +4 16 32 SMD LMD F/D BufferD/A BufferA/M Buffer M/W Buffer Pipelined DLX Datapath Branch Address Calculation Decide Condition Branch Decision for target address

24. HazardsCS510 Computer Architectures Lecture 7 - 24 Control Hazard on Branches: Three Stall Cycles IM Reg ALU DMReg IM Reg ALU DMReg IM Reg ALU DMReg Time(clock cycles) CC1 CC2 CC3 CC4 CC5 CC6 CC7 CC8 CC9 IM Reg ALU DMReg IM Reg ALU DMReg Program execution order in instructions 40 BEQ R1,R3, 36 44 AND R12,R2, R5 48 OR R13,R6, R2 52 ADD R14,R2, R2 80 LD R4,R7, 100 Should’t be executed when branch condition is true ! IM Reg ALU DMReg Branch Delay = 3 cycles IM Reg ALU DMReg IM Reg ALU DMReg Branch Target available

25. HazardsCS510 Computer Architectures Lecture 7 - 25 Control Hazard on Branches: Three Stall Cycles Branch instruction IF ID EX MEM WB Branch successor IF ID EX MEM 3 Wasted clock cycles for the TAKEN branch Now, we know the instruction being executed is a branch. But stall until branch target address is known. Now, target address is available.We don’t know yet the instruction being executed is a branch. Fetch the branch successor. Branch successor + 1 IF ID EX Branch successor + 2 IF ID

26. HazardsCS510 Computer Architectures Lecture 7 - 26 Branch Stall Impact If CPI = 1, 30% branch, Stall 3 cycles => new CPI = 1.9 –Half of the ideal speed Two part solution: –Determine the branch is TAKEN or NOT TAKEN sooner, AND –Compute TAKEN Branch Address(Branch Target) earlier DLX branch tests if register = 0 or 1 DLX Solution: Get New PC earlier Zero test to ID stage - Move Zero test to ID stage Additional ADDER - Additional ADDER to calculate New PC(taken PC) in ID stage - 1 clock cycle penalty for branch in contrast to 3 cycles

27. HazardsCS510 Computer Architectures Lecture 7 - 27 Pipelined DLX Datapath To get target addr. earlier To get the Condition Earlier. Target Address available after ID. When a branch instruction is in Execute stage, Next Address is available here.

28. HazardsCS510 Computer Architectures Lecture 7 - 28

29. HazardsCS510 Computer Architectures Lecture 7 - 29 Branch Behavior in Programs Conditional branch frequencies –integer average --- 14 to 16 % –floating point --- 3 to 12 % Forward and backward taken branches –forward taken --- 60 % –backward taken --- 85 % –the average of all conditional branches ---- 67 %

30. HazardsCS510 Computer Architectures Lecture 7 - 30 4 Branch Hazard Alternatives Stall until branch direction is clear Predict branch NOT TAKEN Predict branch TAKEN Delayed branch

31. HazardsCS510 Computer Architectures Lecture 7 - 31 4 Branch Hazard Alternatives: (1) STALL Stall until branch direction is clear Branch instruction IF ID EX MEM WB 3 cycle penalty Branch instruction IF ID EX MEM WB Branch successor stall IF ID EX MEM WB Branch successor + 1 IF ID EX MEM Branch successor + 2 IF ID Revised DLX pipeline(get the branch address at EX) 1 cycle penalty(Branch Delay Slot) Branch successor stall stall stall IF ID EX MEM Branch successor + 1 IF ID EX Branch successor + 2 IF ID

32. HazardsCS510 Computer Architectures Lecture 7 - 32 4 Branch Hazard Alternatives: (2) Predict Branch “NOT TAKEN” Execute successor instructions in the sequence PC+4 is already calculated, so use it to get the next instruction Flush instructions in the pipeline if branch is actually TAKEN Advantage of late pipeline state update 47% of DLX branches are NOT TAKEN on the average NOT TAKEN branch instruction i IF ID EX MEM WB instruction i+1 IF ID EX MEM WB instruction i+2 IF ID EX MEM WB No penalty TAKEN branch instruction i IF ID EX MEM WB instruction i+1 IF ID EX MEM WB instruction T IF ID EX MEM WB 1 cycle penalty Flush this instruction in progress

33. HazardsCS510 Computer Architectures Lecture 7 - 33 4 Branch Hazard Alternatives: (3) Predict Branch “TAKEN” –53% DLX branches TAKEN on average –Branch target address available after ID in DLX DLX still incurs 1 cycle branch penalty for TAKEN branch Other machines: branch target known before outcome 2 cycle penalty in DLX(1 in other machines). 1 cycle penalty in DLX(0 in other machines) NOT TAKEN instruction i IF ID EX MEM WB Instruction T stall IF Instruction i+1 IF ID EX MEM WB TAKEN branch instruction i IF ID EX MEM WB Instruction T stall IF ID EX MEM WB Instruction T+1 IF ID EX MEM WB TAKEN address not available at this time TAKEN address available

34. HazardsCS510 Computer Architectures Lecture 7 - 34 4 Branch Hazard Alternatives: (4) Delayed Branch Delayed Branch –Delay branch to take place AFTER a successor instruction branch instruction sequential successor 1 sequential successor 2........ sequential successor n branch target if taken –1 slot delayed branch allows proper decision and branch target address in 5 stage DLX pipeline with control hazard improvement Delayed Branch of length n

35. HazardsCS510 Computer Architectures Lecture 7 - 35 Delayed Branch Where to get instructions to fill branch delay slot? –Before branch instruction –From the target address: only valuable when branch TAKEN –From fall through: only valuable when branch NOT TAKEN –Canceling branches allow more slots to be filled Compiler effectiveness for single delayed branch slot: –Fills about 60% of delayed branch slots –About 80% of instructions executed in delayed branch slots are useful in computation –About 50% (60% x 80%) of slots usefully filled

36. HazardsCS510 Computer Architectures Lecture 7 - 36 4 Branch Hazard Alternatives: Delayed Branch From target SUB R4, R5, R6 ADD R1, R2, R3 if R1=0 then Delay slot ADD R1, R2, R3 if R1=0 then SUB R4, R5, R6 - Improve performance when TAKEN(loop) - Must be alright to execute rescheduled instructions if Not Taken - May need duplicate the instruction if it is the target of another branch instr. From fall through ADD R1, R2, R3 if R1=0 then SUB R4, R5, R6 Delay slot ADD R1, R2, R3 if R2=0 then SUB R4, R5, R6 - Improve performance when NOT TAKEN - Must be alright to execute instructions of Taken - Always improve performance - Branch must not depend on rescheduled instructions From before ADD R1, R2, R3 if R2=0 then Delay slot if R2=0 then ADD R1, R2, R3

37. HazardsCS510 Computer Architectures Lecture 7 - 37 Limitations on Delayed Branch Difficulty in finding useful instructions to fill the delayed branch slots Solution - Squashing –Delayed branch associated with a branch prediction –Instructions in the predicted path are executed in the delayed branch slot –If the branch outcome is mispredicted, instructions in the delayed branch slot are squashed(discarded)

38. HazardsCS510 Computer Architectures Lecture 7 - 38 Canceling Branch Used when the delayed branch scheduling, i.e., filling the delay slot cannot be done due to –Restrictions on scheduling instructions at the delay slots –Limitations on the ability to predict whether it will TAKE or NOT TAKE at compile time Instruction includes the direction that the branch was predicted –When the branch behaves as predicted, the instructions in the delay slot are executed –When branch is incorrectly predicted, the instructions in the delay slot are turned into No-OPs Canceling Branch allows to fill the delay slot even if the instruction to be filled in the delay slot does not meet the requirements

39. HazardsCS510 Computer Architectures Lecture 7 - 39 Evaluating Branch Alternatives Stall pipeline3 1+0.14x3=1.42 5/1.42=3.5 1.0 Predict Taken1 1+0.14x1=1.14 5/1.14=4.4 1.26 Predict Not Taken1 1+0.14x0.65=1.09 5/1.09=4.5 1.29 Delayed branch0.5 1+0.14x0.5=1.07 5/1.07=4.6 1.31 Pipeline speedup = Pipeline depth / CPI = Pipeline depth 1 + Branch frequency x Branch penalty Conditional and Unconditional collectively 14% frequency, 65% of branch is TAKEN Scheduling Branch CPI speedup vs speedup vs scheme penalty unpipelined stall

40. HazardsCS510 Computer Architectures Lecture 7 - 40 Static(Compiler) Prediction of Taken/Untaken Branches Code Motion LWR1, 0(R2) SUB R1, R1, R3 BEQZR1, L ORR4, R5, R6 ADDR10,R4,R3 L: ADDR7, R8, R9 NOT TAKEN If branch is almost always NOT TAKEN, and R4 is not needed on the taken path, and R5 and R6 are not modified in the following instruction(s), this move can increase speed Depend on LW, need to stall TAKEN If branch is almost always TAKEN, and R7 is not needed, and R8 and R9 are not modified on the fall-through path, this move can increase speed

41. HazardsCS510 Computer Architectures Lecture 7 - 41 Static(Compiler) Prediction of Taken/Untaken Branches Improves strategy for placing instructions in delay slot Two strategies –Direction-based Prediction: TAKEN backward branch, NOT TAKEN forward branch –Profile-based prediction: Record branch behaviors, predict branch based on the prior run(s) Frequency of Misprediction 0% 10% 20% 30% 40% 50% 60% 70% alvinn compress doduc espresso gcc hydro2d mdljsp2 ora swm256 tomcatv Always taken Misprediction Rate 0% 2% 4% 6% 8% 10% 12% 14% alvinn compress doduc espresso gcc hydro2d mdljsp2 ora swm256 tomcatv Taken backwards Not Taken Forwards

42. HazardsCS510 Computer Architectures Lecture 7 - 42 Evaluating Static Branch Prediction Strategies Misprediction rate ignores frequency of branch Instructions between mispredicted branches is a better metric Instructions per mispredicted branch 1 10 100 1000 10000 100000 alvinn compress doduc espresso gcc hydro2d mdljsp2 ora swm256 tomcatv Profile-basedDirection-based

43. HazardsCS510 Computer Architectures Lecture 7 - 43 Pipelining Summary Just overlap tasks, and easy if tasks are independent Speed Up <= Pipeline Depth; if ideal CPI is 1, then: Speedup = Pipeline Depth 1 + Pipeline stall CPI X Clock Cycle Unpipelined Clock Cycle Pipelined Hazards limit performance on computers: Structural: need more HW resources Data: need forwarding, compiler scheduling Control: Dynamic Prediction, Delayed branch slot, Static(compiler) Prediction