1. Scott Beamer, Instructor
inst.eecs.berkeley.edu/~cs61c CS61C : Machine Structures Lecture #21 CPU Design: Pipelining to Improve Performance
Scott Beamer, Instructor
Greet class

2. Review: Single cycle datapath
5 steps to design a processor
1. Analyze instruction set  datapath requirements
2. Select set of datapath components & establish clock methodology
3. Assemble datapath meeting the requirements
4. Analyze implementation of each instruction to determine setting of control points that effects the register transfer.
5. Assemble the control logic
Control is the hard part
MIPS makes that easier
Instructions same size
Source registers always in same place
Immediates same size, location
Operations always on registers/immediates
Processor
Input
Control
Memory
Datapath
Output

3. An Abstract View of the Critical Path
Critical Path (Load Instruction) =
Delay clock through PC (FFs) +
Instruction Memory’s Access Time +
Register File’s Access Time, +
ALU to Perform a 32-bit Add +
Data Memory Access Time Stable Time for Register File Write
Ideal
Instruction
Memory
Instruction
(Assumes a fast controller) Rd Rs Rt 5 5 5
Instruction
Address A
Now with the clocking methodology back in your mind, we can think about how the critical path of our “abstract” datapath may look like.
One thing to keep in mind about the Register File and Ideal Memory (points to both Instruction and Data) is that the Clock input is a factor ONLY during the write operation.
For read operation, the CLK input is not a factor. The register file and the ideal memory behave as if they are combinational logic.
That is you apply an address to the input, then after certain delay, which we called access time, the output is valid.
We will come back to these points (point to the “behave” bullets) later in this lecture.
But for now, let’s look at this “abstract” datapath’s critical path which occurs when the datapath tries to execute the Load instruction.
The time it takes to execute the load instruction are the sum of:
(a) The PC’s clock-to-Q time.
(b) The instruction memory access time.
(c) The time it takes to read the register file.
(d) The ALU delay in calculating the Data Memory Address.
(e) The time it takes to read the Data Memory.
(f) And finally, the setup time for the register file and clock skew.
+3 = 21 (Y:01)
Data
Addr Rw Ra Rb
ALU 32 32 32
Ideal
Data
Memory PC
Register
File
Next Address B
Data In
clk
clk
clk 32

4. Processor Performance
Can we estimate the clock rate (frequency) of our single-cycle processor? We know:
1 cycle per instruction
lw is the most demanding instruction.
Assume approximate delays for major pieces of the datapath:
Instr. Mem, ALU, Data Mem : 2ns each, regfile 1ns
Instruction execution requires: = 8ns
 125 MHz
What can we do to improve clock rate?
Will this improve performance as well?
We want increases in clock rate to result in programs executing quicker.

5. Ways to Improve Clock Frequency
Smaller Process Size
Smallest feature possible in silicon fabrication
Smaller process is faster because of EE reasons, and is smaller so things are closer
Optimize Logic
Re-arrange CL to be faster
Sometimes more logic can be used to reduce delay
Parallel
Do more at once - later…
Cut Down Length of Critical Path
Inserting registers (pipelining) to break up CL

6. Gotta Do Laundry
Ann, Brian, Cathy, Dave each have one load of clothes to wash, dry, fold, and put away A B C D
Washer takes 30 minutes
Dryer takes 30 minutes
“Folder” takes 30 minutes
“Stasher” takes 30 minutes to put clothes into drawers

7. Sequential laundry takes 8 hours for 4 loads30
Time
6 PM 7 8 9 10 11 12 1
2 AM T a s k O r d e B C D A
Sequential laundry takes 8 hours for 4 loads

8. Pipelined laundry takes 3.5 hours for 4 loads!12
2 AM
6 PM 7 8 9 10 11 1
Time 30 T a s k O r d e B C D A
Pipelined laundry takes hours for 4 loads!

9. Latency: time to completely execute a certain task (delay)
General Definitions
Latency: time to completely execute a certain task (delay)
for example, time to read a sector from disk is disk access time or disk latency
Throughput: amount of work that can be done over a period of time (rate)

10. Pipelining Lessons (1/2)
6 PM 7 8 9
Time B C D A 30 T a s k O r d e
Pipelining doesn’t help latency of single task, it helps throughput of entire workload
Multiple tasks operating simultaneously using different resources
Potential speedup = Number pipe stages
Time to “fill” pipeline and time to “drain” it reduces speedup: 2.3X v. 4X in this example

11. Pipelining Lessons (2/2)
Suppose new Washer takes 20 minutes, new Stasher takes 20 minutes. How much faster is pipeline?
Pipeline rate limited by slowest pipeline stage
Unbalanced lengths of pipe stages reduces speedup
6 PM 7 8 9
Time B C D A 30 T a s k O r d e

12. Steps in Executing MIPS
1) IFtch: Instruction Fetch, Increment PC
2) Dcd: Instruction Decode, Read Registers
3) Exec: Mem-ref: Calculate Address Arith-log: Perform Operation
4) Mem: Load: Read Data from Memory Store: Write Data to Memory
5) WB: Write Data Back to Register

13. Pipelined Execution Representation
Time
IFtch
Dcd
Exec
Mem WB
Every instruction must take same number of steps, also called pipeline “stages”, so some will go idle sometimes

14. Review: Datapath for MIPSPC
instruction
memory +4 rt rs rd
registers
ALU
Data
imm
1. Instruction
Fetch
2. Decode/
Register Read
3. Execute
4. Memory
5. Write Back
Use datapath figure to represent pipeline
IFtch
Dcd
Exec
Mem WB
ALU I$
Reg D$

15. Graphical Pipeline Representation
(In Reg, right half highlight read, left half write)
Time (clock cycles) I n s t r. O r d e I$
ALU
Reg
Reg I$ D$
ALU I$
Reg I$
ALU
Reg D$ I$
Load
Add
Store
Sub Or D$
Reg
ALU
Reg D$
ALU
Reg D$
Reg

16. Nonpipelined Execution:
Example
Suppose 2 ns for memory access, 2 ns for ALU operation, and 1 ns for register file read or write; compute instr rate
Nonpipelined Execution:
lw : IF + Read Reg + ALU + Memory + Write Reg = = 8 ns
add: IF + Read Reg + ALU + Write Reg = = 6 ns (recall 8ns for single-cycle processor)
Pipelined Execution:
Max(IF,Read Reg,ALU,Memory,Write Reg) = 2 ns

17. Midterm Regrades due Wed 8/1 Logisim in lab is now 2.1.6
Administrivia
Assignments
HW7 due 8/2
Proj3 due 8/5
Midterm Regrades due Wed 8/1
Logisim in lab is now 2.1.6
Valerie’s OH on Thursday moved to for this week

18. Pipeline Hazard: Matching socks in later load12
2 AM
6 PM 7 8 9 10 11 1
Time 30 T a s k O r d e B C D A E F
bubble
A depends on D; stall since folder tied up

19. Problems for Pipelining CPUs
Limits to pipelining: Hazards prevent next instruction from executing during its designated clock cycle
Structural hazards: HW cannot support some combination of instructions (single person to fold and put clothes away)
Control hazards: Pipelining of branches causes later instruction fetches to wait for the result of the branch
Data hazards: Instruction depends on result of prior instruction still in the pipeline (missing sock)
These might result in pipeline stalls or “bubbles” in the pipeline.

20. Structural Hazard #1: Single Memory (1/2)
Load
Instr 1
Instr 2
Instr 3
Instr 4
ALU
Reg D$ I n s t r. O r d e
Time (clock cycles)
Read same memory twice in same clock cycle

21. Structural Hazard #1: Single Memory (2/2)
Solution:
infeasible and inefficient to create second memory
(We’ll learn about this more next week)
so simulate this by having two Level 1 Caches (a temporary smaller [of usually most recently used] copy of memory)
have both an L1 Instruction Cache and an L1 Data Cache
need more complex hardware to control when both caches miss

22. Structural Hazard #2: Registers (1/2)sw
Instr 1
Instr 2
Instr 3
Instr 4
ALU
Reg D$ I n s t r. O r d e
Time (clock cycles)
Can we read and write to registers simultaneously?

23. Structural Hazard #2: Registers (2/2)
Two different solutions have been used:
1) RegFile access is VERY fast: takes less than half the time of ALU stage
Write to Registers during first half of each clock cycle
Read from Registers during second half of each clock cycle
2) Build RegFile with independent read and write ports
Result: can perform Read and Write during same clock cycle

24. Consider the following sequence of instructions
Data Hazards (1/2)
Consider the following sequence of instructions
add $t0, $t1, $t2
sub $t4, $t0 ,$t3
and $t5, $t0 ,$t6
or $t7, $t0 ,$t8
xor $t9, $t0 ,$t10

25. Data-flow backward in time are hazards
Data Hazards (2/2)
Data-flow backward in time are hazards
Time (clock cycles) I n s t r. O r d e
add $t0,$t1,$t2 IF
ID/RF EX
MEM WB
ALU I$
Reg D$
sub $t4,$t0,$t3
ALU I$
Reg D$
and $t5,$t0,$t6
ALU I$
Reg D$
or $t7,$t0,$t8 I$
ALU
Reg D$
xor $t9,$t0,$t10
ALU I$
Reg D$

26. Data Hazard Solution: Forwarding
Forward result from one stage to another
add $t0,$t1,$t2 IF
ID/RF EX
MEM WB
ALU I$
Reg D$
sub $t4,$t0,$t3
ALU I$
Reg D$
and $t5,$t0,$t6
ALU I$
Reg D$
or $t7,$t0,$t8 I$
ALU
Reg D$
xor $t9,$t0,$t10
ALU I$
Reg D$
“or” hazard solved by register hardware

27. Dataflow backwards in time are hazards
Data Hazard: Loads (1/4)
Dataflow backwards in time are hazards IF
ID/RF EX
MEM WB
lw $t0,0($t1) I$
Reg
ALU
Reg D$
sub $t3,$t0,$t2
ALU I$
Reg D$
Can’t solve all cases with forwarding
Must stall instruction dependent on load, then forward (more hardware)

28. Hardware stalls pipeline Called “interlock”
Data Hazard: Loads (2/4)
Hardware stalls pipeline
Called “interlock”
lw $t0, 0($t1) IF
ID/RF EX
MEM WB
ALU I$
Reg D$
sub $t3,$t0,$t2
ALU I$
Reg D$
bubble
and $t5,$t0,$t4
ALU I$
Reg D$
bubble
or $t7,$t0,$t6 I$
ALU
Reg D$
bubble

29. Data Hazard: Loads (3/4)
Instruction slot after a load is called “load delay slot”
If that instruction uses the result of the load, then the hardware interlock will stall it for one cycle.
If the compiler puts an unrelated instruction in that slot, then no stall
Letting the hardware stall the instruction in the delay slot is equivalent to putting a nop in the slot (except the latter uses more code space)

30. Stall is equivalent to nop
Data Hazard: Loads (4/4)
Stall is equivalent to nop
lw $t0, 0($t1)
ALU I$
Reg D$
bubble
nop
ALU I$
Reg D$
sub $t3,$t0,$t2
ALU I$
Reg D$
and $t5,$t0,$t4
or $t7,$t0,$t6 I$
ALU
Reg D$

31. First MIPS design did not interlock and stall on load-use data hazard
Historical Trivia
First MIPS design did not interlock and stall on load-use data hazard
Real reason for name behind MIPS: Microprocessor without Interlocked Pipeline Stages
Word Play on acronym for Millions of Instructions Per Second, also called MIPS

32. Peer Instruction
Thanks to pipelining, I have reduced the time it took me to wash my shirt.
Longer pipelines are always a win (since less work per stage & a faster clock).
We can rely on compilers to help us avoid data hazards by reordering instrs.
ABC
0: FFF
1: FFT
2: FTF
3: FTT
4: TFF
5: TFT
6: TTF
7: TTT

33. Peer Instruction Answer
Throughput better, not execution time
“…longer pipelines do usually mean faster clock, but branches cause problems!
“they happen too often & delay too long.” Forwarding! (e.g, Mem  ALU)
F A L S E
Thanks to pipelining, I have reduced the time it took me to wash my shirt.
Longer pipelines are always a win (since less work per stage & a faster clock).
We can rely on compilers to help us avoid data hazards by reordering instrs.
ABC
0: FFF
1: FFT
2: FTF
3: FTT
4: TFF
5: TFT
6: TTF
7: TTT
F A L S E
F A L S E

34. Things to Remember Optimal Pipeline What makes this work?
Each stage is executing part of an instruction each clock cycle.
One instruction finishes during each clock cycle.
On average, execute far more quickly.
What makes this work?
Similarities between instructions allow us to use same stages for all instructions (generally).
Each stage takes about the same amount of time as all others: little wasted time.