CS 152 Computer Architecture and Engineering Lecture 4 - Pipelining Krste Asanovic Electrical Engineering and Computer Sciences University of California at Berkeley http://www.eecs.berkeley.edu/~krste http://inst.eecs.berkeley.edu/~cs152 CS252 S05

Last time in Lecture 3 Microcoding became less attractive as gap between RAM and ROM speeds reduced Complex instruction sets difficult to pipeline, so difficult to increase performance as gate count grew Iron-law explains architecture design space Trade instruction/program, cycles/instruction, and time/cycle Load-Store RISC ISAs designed for efficient pipelined implementations Very similar to vertical microcode Inspired by earlier Cray machines 2/5/2008 CS152-Spring’08 CS252 S05

“Iron Law” of Processor Performance Time = Instructions Cycles Time Program Program * Instruction * Cycle Instructions per program depends on source code, compiler technology, and ISA Cycles per instructions (CPI) depends upon the ISA and the microarchitecture Time per cycle depends upon the microarchitecture and the base technology Microarchitecture CPI cycle time Microcoded >1 short Single-cycle unpipelined 1 long Pipelined Lecture 3 This lecture 2/5/2008 CS152-Spring’08 CS252 S05

An Ideal Pipeline All objects go through the same stages 1 2 3 4 All objects go through the same stages No sharing of resources between any two stages Propagation delay through all pipeline stages is equal The scheduling of an object entering the pipeline is not affected by the objects in other stages These conditions generally hold for industrial assembly lines. But can an instruction pipeline satisfy the last condition? 2/5/2008 CS152-Spring’08 CS252 S05

Pipelined MIPS To pipeline MIPS: First build MIPS without pipelining with CPI=1 Next, add pipeline registers to reduce cycle time while maintaining CPI=1 2/5/2008 CS152-Spring’08 CS252 S05

Pipelined Datapath write -back phase fetch execute decode & Reg-fetch IR PC Add we rs1 rs2 rd1 addr we rdata ws addr wd ALU rd2 GPRs rdata Data Memory Inst. Memory Imm Ext wdata write -back phase fetch execute decode & Reg-fetch memory Clock period can be reduced by dividing the execution of an instruction into multiple cycles tC > max {tIM, tRF, tALU, tDM, tRW} ( = tDM probably) However, CPI will increase unless instructions are pipelined 2/5/2008 CS152-Spring’08 CS252 S05

Technology Assumptions A small amount of very fast memory (caches) backed up by a large, slower memory Fast ALU (at least for integers) Multiported Register files (slower!) Thus, the following timing assumption is reasonable tIM tRF tALU tDM  tRW A 5-stage pipelined Harvard architecture will be the focus of our detailed design 2/5/2008 CS152-Spring’08 CS252 S05

5-Stage Pipelined Execution Write -Back (WB) I-Fetch (IF) Execute (EX) Decode, Reg. Fetch (ID) Memory (MA) addr wdata rdata Data Memory we ALU Imm Ext 0x4 Add Inst. rd1 GPRs rs1 rs2 ws wd rd2 IR PC time t0 t1 t2 t3 t4 t5 t6 t7 . . . . instruction1 IF1 ID1 EX1 MA1 WB1 instruction2 IF2 ID2 EX2 MA2 WB2 instruction3 IF3 ID3 EX3 MA3 WB3 instruction4 IF4 ID4 EX4 MA4 WB4 instruction5 IF5 ID5 EX5 MA5 WB5 2/5/2008 CS152-Spring’08 CS252 S05

5-Stage Pipelined Execution Resource Usage Diagram Write -Back (WB) I-Fetch (IF) Execute (EX) Decode, Reg. Fetch (ID) Memory (MA) addr wdata rdata Data Memory we ALU Imm Ext 0x4 Add Inst. rd1 GPRs rs1 rs2 ws wd rd2 IR PC time t0 t1 t2 t3 t4 t5 t6 t7 . . . . IF I1 I2 I3 I4 I5 ID I1 I2 I3 I4 I5 EX I1 I2 I3 I4 I5 MA I1 I2 I3 I4 I5 WB I1 I2 I3 I4 I5 Resources 2/5/2008 CS152-Spring’08 CS252 S05

Pipelined Execution: ALU Instructions PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add IR Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata rdata Data IR 31 Not quite correct! We need an Instruction Reg (IR) for each stage 2/5/2008 CS152-Spring’08 CS252 S05

Pipelined MIPS Datapath without jumps F D E M W IR WBSrc MemWrite 31 OpSel 0x4 Add RegDst RegWrite rd1 GPRs rs1 rs2 ws wd rd2 we PC A addr inst Inst Memory wdata addr rdata we IR Y ALU B Data Memory R ExtSel BSrc Imm Ext MD1 MD2 Control Points Need to Be Connected 2/5/2008 CS152-Spring’08 CS252 S05

How Instructions can Interact with each other in a pipeline An instruction in the pipeline may need a resource being used by another instruction in the pipeline  structural hazard An instruction may depend on something produced by an earlier instruction Dependence may be for a data value  data hazard Dependence may be for the next instruction’s address  control hazard (branches, exceptions) 2/5/2008 CS152-Spring’08 CS252 S05

Data Hazards r1 is stale. Oops! r4  r1 … r1 … ... r1 r0 + 10 IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata rdata Data ... r1 r0 + 10 r4 r1 + 17 r1 is stale. Oops! 2/5/2008 CS152-Spring’08 CS252 S05

Resolving Data Hazards (1) Strategy 1: Wait for the result to be available by freezing earlier pipeline stages  interlocks 2/5/2008 CS152-Spring’08 CS252 S05

Feedback to Resolve Hazards FB1 FB2 FB3 FB4 stage 1 2 3 4 Later stages provide dependence information to earlier stages which can stall (or kill) instructions Real designs will seldom provide full feedback nor will they be able to stop on a dime. Controlling a pipeline in this manner works provided the instruction at stage i+1 can complete without any interference from instructions in stages 1 to i (otherwise deadlocks may occur) 2/5/2008 CS152-Spring’08 CS252 S05

Interlocks to resolve Data Hazards Stall Condition IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata rdata Data nop ... r1 r0 + 10 r4 r1 + 17 2/5/2008 CS152-Spring’08 CS252 S05

Stalled Stages and Pipeline Bubbles time t0 t1 t2 t3 t4 t5 t6 t7 . . . . (I1) r1 (r0) + 10 IF1 ID1 EX1 MA1 WB1 (I2) r4 (r1) + 17 IF2 ID2 ID2 ID2 ID2 EX2 MA2 WB2 (I3) IF3 IF3 IF3 IF3 ID3 EX3 MA3 WB3 (I4) IF4 ID4 EX4 MA4 WB4 (I5) IF5 ID5 EX5 MA5 WB5 stalled stages time t0 t1 t2 t3 t4 t5 t6 t7 . . . . IF I1 I2 I3 I3 I3 I3 I4 I5 ID I1 I2 I2 I2 I2 I3 I4 I5 EX I1 nop nop nop I2 I3 I4 I5 MA I1 nop nop nop I2 I3 I4 I5 WB I1 nop nop nop I2 I3 I4 I5 Resource Usage nop  pipeline bubble 2/5/2008 CS152-Spring’08 CS252 S05

Interlock Control Logic Cstall ws rs rt ? stall IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata rdata Data nop Compare the source registers of the instruction in the decode stage with the destination register of the uncommitted instructions. 2/5/2008 CS152-Spring’08 CS252 S05

Interlock Control Logic ignoring jumps & branches IR PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata rdata Data 31 nop stall Cstall rs rt ? we ws we Cdest re1 re2 Cre Cdest Should we always stall if the rs field matches some rd? not every instruction writes a register we not every instruction reads a register re 2/5/2008 CS152-Spring’08 CS252 S05

Source & Destination Registers R-type: op rs rt rd func I-type: op rs rt immediate16 J-type: op immediate26 source(s) destination ALU rd  (rs) func (rt) rs, rt rd ALUi rt  (rs) op imm rs rt LW rt M [(rs) + imm] rs rt SW M [(rs) + imm]  (rt) rs, rt BZ cond (rs) true: PC  (PC) + imm rs false: PC  (PC) + 4 rs J PC  (PC) + imm JAL r31  (PC), PC  (PC) + imm 31 JR PC  (rs) rs JALR r31  (PC), PC  (rs) rs 31 2/5/2008 CS152-Spring’08 CS252 S05

Deriving the Stall Signal Cdest ws = Case opcode ALU rd ALUi, LW rt JAL, JALR R31 we = Case opcode ALU, ALUi, LW (ws  0) JAL, JALR on ... off Cre re1 = Case opcode ALU, ALUi, on off re2 = Case opcode LW, SW, BZ, JR, JALR J, JAL ALU, SW ... Cstall stall = ((rsD =wsE).weE + (rsD =wsM).weM + (rsD =wsW).weW) . re1D + ((rtD =wsE).weE + (rtD =wsM).weM + (rtD =wsW).weW) . re2D This is not the full story ! 2/5/2008 CS152-Spring’08 CS252 S05

Hazards due to Loads & Stores IR 31 PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata rdata Data nop Stall Condition What if (r1)+7 = (r3)+5 ? ... M[(r1)+7]  (r2) r4  M[(r3)+5] Is there any possible data hazard in this instruction sequence? 2/5/2008 CS152-Spring’08 CS252 S05

(r1)+7 = (r3)+5  data hazard Load & Store Hazards ... M[(r1)+7]  (r2) r4  M[(r3)+5] (r1)+7 = (r3)+5  data hazard However, the hazard is avoided because our memory system completes writes in one cycle ! Load/Store hazards are sometimes resolved in the pipeline and sometimes in the memory system itself. More on this later in the course. 2/5/2008 CS152-Spring’08 CS252 S05

CS152 Administrivia Krste, office hours, Monday 1-3pm, 645 Soda email for alternate time Henry office hours, 511 Soda 9:30-10:30AM Mondays 2:00-3:00PM Fridays First lab and practice problems this evening In-class quiz dates: Q1: Tuesday February 19 (ISAs, microcode, simple pipelining) Q2: Tuesday March 4 (memory hierarchies) 2/5/2008 CS152-Spring’08 CS252 S05

Course Schedule Check website for schedule Generally, a lab, practice problem set, plus quiz every two weeks Lab and practice problems out on Tuesday (this evening for first) Lab overview in section next day (Wednesday) Practice problems due following Tuesday in class, solutions available on line Problem review in section next day (Wednesday) Lab due in class Thursday Quiz following Tuesday, with new PS and lab. 2/5/2008 CS152-Spring’08

Resolving Data Hazards (2) Strategy 2: Route data as soon as possible after it is calculated to the earlier pipeline stage  bypass 2/5/2008 CS152-Spring’08 CS252 S05

Bypassing Each stall or kill introduces a bubble in the pipeline time t0 t1 t2 t3 t4 t5 t6 t7 . . . . (I1) r1 r0 + 10 IF1 ID1 EX1 MA1 WB1 (I2) r4 r1 + 17 IF2 ID2 ID2 ID2 ID2 EX2 MA2 WB2 (I3) IF3 IF3 IF3 IF3 ID3 EX3 MA3 (I4) stalled stages IF4 ID4 EX4 (I5) IF5 ID5 Each stall or kill introduces a bubble in the pipeline  CPI > 1 A new datapath, i.e., a bypass, can get the data from the output of the ALU to its input time t0 t1 t2 t3 t4 t5 t6 t7 . . . . (I1) r1 r0 + 10 IF1 ID1 EX1 MA1 WB1 (I2) r4  r1 + 17 IF2 ID2 EX2 MA2 WB2 (I3) IF3 ID3 EX3 MA3 WB3 (I4) IF4 ID4 EX4 MA4 WB4 (I5) IF5 ID5 EX5 MA5 WB5 2/5/2008 CS152-Spring’08 CS252 S05

Adding a Bypass ... (I1) r1 r0 + 10 (I2) r4 r1 + 17 r4  r1... PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add Imm Ext ALU rd1 GPRs rs1 rs2 ws wd rd2 we wdata rdata Data 31 nop stall D E M W ... (I1) r1 r0 + 10 (I2) r4 r1 + 17 r4  r1... r1 ... ASrc When does this bypass help? r1 M[r0 + 10] r4 r1 + 17 JAL 500 r4 r31 + 17 yes no no 2/5/2008 CS152-Spring’08 CS252 S05

The Bypass Signal Deriving it from the Stall Signal stall = ( ((rsD =wsE).weE + (rsD =wsM).weM + (rsD =wsW).weW).re1D +((rtD =wsE).weE + (rtD =wsM).weM + (rtD =wsW).weW).re2D ) ws = Case opcode ALU rd ALUi, LW rt JAL, JALR R31 we = Case opcode ALU, ALUi, LW (ws  0) JAL, JALR on ... off ASrc = (rsD=wsE).weE.re1D Is this correct? No because only ALU and ALUi instructions can benefit from this bypass We can’t bypass on memory or JAL* instructions. Split weE into two components: we-bypass, we-stall 2/5/2008 CS152-Spring’08 CS252 S05

Bypass and Stall Signals Split weE into two components: we-bypass, we-stall we-bypassE = Case opcodeE ALU, ALUi  (ws  0) ... off we-stallE = Case opcodeE LW  (ws  0) JAL, JALR on ... off ASrc = (rsD =wsE).we-bypassE . re1D stall = ((rsD =wsE).we-stallE + (rsD=wsM).weM + (rsD=wsW).weW). re1D +((rtD = wsE).weE + (rtD = wsM).weM + (rtD = wsW).weW). re2D 2/5/2008 CS152-Spring’08 CS252 S05

Fully Bypassed Datapath ASrc IR PC A B Y R MD1 MD2 addr inst Inst Memory 0x4 Add ALU Imm Ext rd1 GPRs rs1 rs2 ws wd rd2 we wdata rdata Data 31 nop stall D E M W PC for JAL, ... BSrc Is there still a need for the stall signal ? stall = (rsD=wsE). (opcodeE=LWE).(wsE0 ).re1D + (rtD=wsE). (opcodeE=LWE).(wsE0 ).re2D 2/5/2008 CS152-Spring’08 CS252 S05

Resolving Data Hazards (3) Strategy 3: Speculate on the dependence. Two cases: Guessed correctly  do nothing Guessed incorrectly  kill and restart 2/5/2008 CS152-Spring’08 CS252 S05

Instruction to Instruction Dependence What do we need to calculate next PC: For Jumps Opcode, offset and PC For Jump Register Opcode and Register value For Conditional Branches Opcode, PC, Register (for condition), and offset For all others Opcode and PC 2/5/2008 CS152-Spring’08 CS252 S05

PC Calculation Bubbles time t0 t1 t2 t3 t4 t5 t6 t7 . . . . (I1) r1 (r0) + 10 IF1 ID1 EX1 MA1 WB1 (I2) r3 (r2) + 17 IF2 IF2 ID2 EX2 MA2 WB2 (I3) IF3 IF3 ID3 EX3 MA3 WB3 (I4) IF4 IF4 ID4 EX4 MA4 WB4 time t0 t1 t2 t3 t4 t5 t6 t7 . . . . IF I1 nop I2 nop I3 nop I4 ID I1 nop I2 nop I3 nop I4 EX I1 nop I2 nop I3 nop I4 MA I1 nop I2 nop I3 nop I4 WB I1 nop I2 nop I3 nop I4 Resource Usage nop  pipeline bubble 2/5/2008 CS152-Spring’08 CS252 S05

Speculate next address is PC+4 104 IR PC addr inst Inst Memory 0x4 Add nop E M Jump? PCSrc (pc+4 / jabs / rind/ br) stall I1 096 ADD I2 100 J 304 I3 104 ADD I4 304 ADD A jump instruction kills (not stalls) the following instruction kill How? 2/5/2008 CS152-Spring’08 CS252 S05

Pipelining Jumps PCSrc (pc+4 / jabs / rind/ br) stall To kill a fetched instruction -- Insert a mux before IR Add E M 0x4 nop IR IR Add Jump? I2 I1 304 nop I2 I1 104 Any interaction between stall and jump? nop IRSrcD addr PC inst IR Kill takes precedence over stall. Inst Memory IRSrcD = Case opcodeD J, JAL nop ... IM I1 096 ADD I2 100 J 304 I3 104 ADD I4 304 ADD kill 2/5/2008 CS152-Spring’08 CS252 S05

Jump Pipeline Diagrams time t0 t1 t2 t3 t4 t5 t6 t7 . . . . (I1) 096: ADD IF1 ID1 EX1 MA1 WB1 (I2) 100: J 304 IF2 ID2 EX2 MA2 WB2 (I3) 104: ADD IF3 nop nop nop nop (I4) 304: ADD IF4 ID4 EX4 MA4 WB4 time t0 t1 t2 t3 t4 t5 t6 t7 . . . . IF I1 I2 I3 I4 I5 ID I1 I2 nop I4 I5 EX I1 I2 nop I4 I5 MA I1 I2 nop I4 I5 WB I1 I2 nop I4 I5 Resource Usage nop  pipeline bubble 2/5/2008 CS152-Spring’08 CS252 S05

Pipelining Conditional Branches 104 stall IR PC addr inst Inst Memory 0x4 Add nop E M PCSrc (pc+4 / jabs / rind / br) IRSrcD BEQZ? A Y ALU zero? Branch condition is not known until the execute stage what action should be taken in the decode stage ? I1 096 ADD I2 100 BEQZ r1 +200 I3 104 ADD I4 304 ADD 2/5/2008 CS152-Spring’08 CS252 S05

Pipelining Conditional Branches PCSrc (pc+4 / jabs / rind / br) stall BEQZ? ? Add E M 0x4 nop A Y ALU zero? IR IR Add I2 I1 108 I3 nop IRSrcD addr PC inst IR Inst Memory If the branch is taken - kill the two following instructions - the instruction at the decode stage is not valid  stall signal is not valid I1 096 ADD I2 100 BEQZ r1 +200 I3 104 ADD I4 304 ADD 2/5/2008 CS152-Spring’08 CS252 S05

Pipelining Conditional Branches PCSrc (pc+4/jabs/rind/br) stall Add PC BEQZ? E M IRSrcE 0x4 nop A Y ALU zero? IR IR Add Jump? I2 I1 108 I3 IRSrcD addr PC nop inst IR Inst Memory If the branch is taken - kill the two following instructions - the instruction at the decode stage is not valid  stall signal is not valid I1 096 ADD I2 100 BEQZ r1 200 I3 104 ADD I4 304 ADD 2/5/2008 CS152-Spring’08 CS252 S05

New Stall Signal Don’t stall if the branch is taken. Why? stall = ( ((rsD =wsE).weE + (rsD =wsM).weM + (rsD =wsW).weW).re1D + ((rtD =wsE).weE + (rtD =wsM).weM + (rtD =wsW).weW).re2D ) . !((opcodeE=BEQZ).z + (opcodeE=BNEZ).!z) Don’t stall if the branch is taken. Why? Instruction at the decode stage is invalid 2/5/2008 CS152-Spring’08 CS252 S05

Control Equations for PC and IR Muxes PCSrc = Case opcodeE BEQZ.z, BNEZ.!z br ...  Case opcodeD J, JAL  jabs JR, JALR  rind ...  pc+4 Give priority to the older instruction, i.e., execute stage instruction over decode IRSrcD = Case opcodeE BEQZ.z, BNEZ.!z nop ...  Case opcodeD J, JAL, JR, JALR nop ... IM IRSrcE = Case opcodeE BEQZ.z, BNEZ.!z nop ... stall.nop + !stall.IRD 2/5/2008 CS152-Spring’08 CS252 S05

Branch Pipeline Diagrams (resolved in execute stage) time t0 t1 t2 t3 t4 t5 t6 t7 . . . . (I1) 096: ADD IF1 ID1 EX1 MA1 WB1 (I2) 100: BEQZ 200 IF2 ID2 EX2 MA2 WB2 (I3) 104: ADD IF3 ID3 nop nop nop (I4) 108: IF4 nop nop nop nop (I5) 304: ADD IF5 ID5 EX5 MA5 WB5 time t0 t1 t2 t3 t4 t5 t6 t7 . . . . IF I1 I2 I3 I4 I5 ID I1 I2 I3 nop I5 EX I1 I2 nop nop I5 MA I1 I2 nop nop I5 WB I1 I2 nop nop I5 Resource Usage nop  pipeline bubble 2/5/2008 CS152-Spring’08 CS252 S05

Reducing Branch Penalty (resolve in decode stage) One pipeline bubble can be removed if an extra comparator is used in the Decode stage PCSrc (pc+4 / jabs / rind/ br) Add IR nop E 0x4 Add Zero detect on register file output rd1 GPRs rs1 rs2 ws wd rd2 we nop PC addr inst IR Inst Memory D Pipeline diagram now same as for jumps 2/5/2008 CS152-Spring’08 CS252 S05

Branch Delay Slots (expose control hazard to software) Change the ISA semantics so that the instruction that follows a jump or branch is always executed gives compiler the flexibility to put in a useful instruction where normally a pipeline bubble would have resulted. I1 096 ADD I2 100 BEQZ r1 200 I3 104 ADD I4 304 ADD Delay slot instruction executed regardless of branch outcome Other techniques include branch prediction, which can dramatically reduce the branch penalty... to come later 2/5/2008 CS152-Spring’08 CS252 S05

Why an Instruction may not be dispatched every cycle (CPI>1) Full bypassing may be too expensive to implement typically all frequently used paths are provided some infrequently used bypass paths may increase cycle time and counteract the benefit of reducing CPI Loads have two cycle latency Instruction after load cannot use load result MIPS-I ISA defined load delay slots, a software-visible pipeline hazard (compiler schedules independent instruction or inserts NOP to avoid hazard). Removed in MIPS-II. Conditional branches may cause bubbles kill following instruction(s) if no delay slots Machines with software-visible delay slots may execute significant number of NOP instructions inserted by the compiler. 2/5/2008 CS152-Spring’08 CS252 S05

Acknowledgements These slides contain material developed and copyright by: Arvind (MIT) Krste Asanovic (MIT/UCB) Joel Emer (Intel/MIT) James Hoe (CMU) John Kubiatowicz (UCB) David Patterson (UCB) MIT material derived from course 6.823 UCB material derived from course CS252 2/5/2008 CS152-Spring’08 CS252 S05