1. CSCI-365 Computer Organization Lecture Note: Some slides and/or pictures in the following are adapted from: Computer Organization and Design, Patterson & Hennessy, ©2005

2. Recap: Single Cycle Datapath Uses the clock cycle inefficiently – the clock cycle must be timed to accommodate the slowest instruction –especially problematic for more complex instructions like floating point multiply Clk lwsw Waste Cycle 1Cycle 2

3. Instruction Times (Critical Paths) Instr.I MemReg RdALU OpD MemReg WrTotal R-type(45%) Load (25%) Store (10%) Beq (15%) Jump (5%)  What is the clock cycle time assuming negligible delays for muxes, control unit, sign extend, PC access, shift left 2, wires, setup and hold times except: l Instruction and Data Memory (200 ps) l ALU and adders (200 ps) l Register File access (reads or writes) (100 ps)

4. Simplified MIPS Pipelined Datapath

5. Instruction Critical Paths Instr.I MemReg RdALU OpD MemReg WrTotal R- type load store beq jump 200100200100600 200100200 100800  What is the clock cycle time assuming negligible delays for muxes, control unit, sign extend, PC access, shift left 2, wires, setup and hold times except: l Instruction and Data Memory (200 ps) l ALU and adders (200 ps) l Register File access (reads or writes) (100 ps) 200100200 700 200100200500 200

6. Stages Five stages, one step per stage 1.IF: Instruction fetch from memory 2.ID: Instruction decode & register read 3.EX: Execute operation or calculate address 4.MEM: Access memory operand 5.WB: Write result back to register

7. Single Cycle vs. Multiple Cycle Multiple Cycle Implementation: Clk Cycle 1 IFIDEXMEMWB Cycle 2Cycle 3Cycle 4Cycle 5Cycle 6Cycle 7Cycle 8Cycle 9Cycle 10 IFIDEXMEM lwsw IF R-type Clk Single Cycle Implementation: lwsw Waste Cycle 1Cycle 2

8. Gotta Do Laundry Michael, Conan, Jimmy, Pat each have one load of clothes to wash, dry, fold, and put away –Washer takes 30 minutes –Dryer takes 30 minutes –“Folder” takes 30 minutes –“Stasher” takes 30 minutes to put clothes into drawers MCJP

9. Sequential laundry takes 8 hours for 4 loads TaskOrderTaskOrder C J P M 30 Time 30 6 PM 7 8 9 10 11 12 1 2 AM Sequential Laundry

10. Pipelined laundry takes 3.5 hours for 4 loads! TaskOrderTaskOrder C J P M 12 2 AM 6 PM 7 8 9 10 11 1 Time 30 Pipelined Laundry

11. General Definitions Latency: time to completely execute a certain task –E.g., time to read a sector from disk is disk access time or disk latency Throughput: amount of work that can be done over a period of time

12. Pipelining doesn’t help latency of single task, it helps throughput of entire workload Multiple tasks operating simultaneously using different resources Potential speedup = Number pipe stages Time to “fill” pipeline and time to “drain” it reduces speedup: 2.3X v. 4X in this example 6 PM 789 Time C J P M 30 TaskOrderTaskOrder Pipelining Lessons

13. Suppose new Washer takes 20 minutes, new Stasher takes 20 minutes. How much faster is pipeline? –Pipeline rate limited by slowest pipeline stage –Unbalanced lengths of pipe stages reduces speedup

14. A Pipelined MIPS Processor Start the next instruction before the current one has completed –improves throughput –instruction latency is not reduced –clock cycle (pipeline stage time) limited by slowest stage –for some instructions, some stages are wasted cycles Cycle 1Cycle 2Cycle 3Cycle 4Cycle 5 IFIDEXMEMWB lw Cycle 7Cycle 6Cycle 8 sw IFIDEXMEMWB R-type IFIDEXMEMWB

15. Single Cycle vs. Multiple Cycle vs. Pipelined Multiple Cycle Implementation: Clk Cycle 1 IFIDEXMEMWB Cycle 2Cycle 3Cycle 4Cycle 5Cycle 6Cycle 7Cycle 8Cycle 9Cycle 10 IFIDEXMEM lwsw IF R-type Clk Single Cycle Implementation: lwsw Waste Cycle 1Cycle 2 lw IFIDEXMEMWB Pipeline Implementation: IFIDEXMEMWB sw IFIDEXMEMWB R-type

16. Single Cycle vs. Pipelined Example: Compare average time between lw instructions of a single cycle implementation to a pipelined implementation. Assume following operation times for major functional units –200 ps for memory access –200 ps for ALU operation –100 ps for register file read or write (DONE IN CLASS, try 3 and 100, and n)

17. Pipelined Control (Simplified)

18. Pipelined Control

19. Simplified MIPS Pipelined Datapath Can you foresee any problems with these right-to-left flows?

20. Pipeline registers Need registers between stages –To hold information produced in previous cycle

21. IF

22. ID

23. EX for Load

24. MEM for Load

25. WB for Load Wrong register number There is a BUG here

26. Corrected Datapath for Load

27. Pipelined Control Control signals derived from instruction –As in single-cycle implementation

28. Hazards Situations that prevent starting the next instruction in the next cycle –Structure hazards –Data hazard –Control hazard

29. Structure Hazards Instruction cannot execute in proper clock cycle because hardware cannot support the combination of instructions that are set to execute in the clock cycle In MIPS pipeline with a single memory –Load/store requires data access –Instruction fetch would have to stall for that cycle Would cause a pipeline “bubble” Hence, pipelined datapaths require separate instruction/data memories –Or separate instruction/data caches

30. Data Hazards Instruction cannot execute in proper clock cycle because data that is needed is not yet available add$s0, $t0, $t1 sub$t2, $s0, $t3

31. Simplified MIPS Pipelined Datapath

32. Forwarding (aka Bypassing) Use result when it is computed –Don’t wait for it to be stored in a register –Requires extra connections in the datapath

33. Load-Use Data Hazard Can’t always avoid stalls by forwarding –If value not computed when needed –Can’t forward backward in time!

34. Code Scheduling to Avoid Stalls Reorder code to avoid use of load result in the next instruction: C code A = B + E, C= B + F lw$t1, 0($t0) lw$t2, 4($t0) add$t3, $t1, $t2 sw$t3, 12($t0) lw$t4, 8($t0) add$t5, $t1, $t4 sw$t5, 16($t0) stall lw$t1, 0($t0) lw$t2, 4($t0) lw$t4, 8($t0) add$t3, $t1, $t2 sw$t3, 12($t0) add$t5, $t1, $t4 sw$t5, 16($t0)

35. Control Hazards Instruction cannot execute in proper clock cycle because the instruction that was fetched is not the one that is needed –Branch determines flow of control –Fetching next instruction depends on branch outcome –Pipeline can’t always fetch correct instruction In MIPS pipeline –Need to compare registers and compute target early in the pipeline –Add hardware to do it in ID stage

36. Stall on Branch Wait until branch outcome determined before fetching next instruction

37. Branch Prediction Correct branch prediction is very important and can produce substantial performance improvements. –static prediction –dynamic prediction To take full advantage of branch prediction, we can have the instructions not only fetched but also begin execution. This is known as speculative execution

38. MIPS with Predict Not Taken Prediction correct Prediction incorrect

39. More Realistic Branch Prediction Static branch prediction –Based on typical branch behavior –Example: loop and if-statement branches Predict backward branches taken Predict forward branches not taken Dynamic branch prediction –Hardware measures actual branch behavior e.g., record recent history of each branch –Assume future behavior will continue the trend When wrong, stall while re-fetching, and update history

40. Branches Branch instructions can dramatically affect pipeline performance. Control operations are very frequent in current programs. 20% - 35% of the instructions executed are branches (conditional and unconditional). 65% of the branches actually take the branch. Conditional branches are much more frequent than unconditional (more than two times). More than 50% of conditional branches are taken.

41. Static Branch Prediction Static prediction techniques do not take into consideration execution history. Predict never taken (Motorola 68020): assumes that the branch is not taken. Predict always taken: assumes that the branch is taken.

42. Dynamic Branch Prediction Improve the accuracy of prediction by recording the history of conditional branches. One-bit prediction scheme –is used in order to record if the last execution resulted in a branch taken or not. The system predicts the same behavior as for the last time. Two-bit prediction scheme –with a two-bit scheme predictions can be made depending on the last two instances of execution.

43. One-Bit Prediction Scheme

44. Two-Bit Prediction Scheme

45. Branch History Table History info. can be used not only to predict the outcome of a conditional branch but also to avoid recalculation of the target address. Together with bits used for prediction, the target address can be stored for later use in a branch history table. Using D. B. P with history tables up to 90% of predictions can be correct. Pentium,PowerPC620 use speculative execution with D.B.P based on a branch history table.

46. Branch History Table