1. ECE 361 Computer Architecture Lecture 13: Designing a Pipeline Processor
Start X:40

2. Review: A Pipelined Datapath
Clk
Ifetch
Reg/Dec
Exec
Mem Wr
RegWr
ExtOp
ALUOp
Branch 1
PC+4
PC+4
The pipelined datapath consists of combination logic blocks separated by pipeline registers. If you get rid of all these registers (not the PC), this pipelined datapath is reduced to the single-cycle datapath.
This should give you extra incentive to do a good job on your single cycle processor design homework because you can build your pipeline design based on your single cycle design.
Anyway, the registers mark the beginning and the end of a pipe stage.
In the multiple clock cycle lecture, I recommended that the best way to think about a logic clock cycle is that it begins slightly after the clock tick and ends right at the next clock tick.
For example here, the Reg/Decode stage begins slightly after this clock tick when the output of the IF/ID register has stabilized to its new value AND ends RIGHT at the next clock tick when the output of the register file is clocked into the ID/Exec register.
At the end of the Reg/Decode stage, the register output that just clocked into the ID/Exec register has NOT yet propagate to the register output yet. It takes a Clk-to-Q delay.
When the new value we just clocked in (points to the clock tick) has propagate to the register output, then we have reach the beginning of the Exec stage.
Notice that the Wr stage of the pipeline starts here (last cycle) but there is no corresponding datapath underneath it because the Wr stage of the pipeline is handled by the same part of the pipeline that handles the Register Read stage.
This part of the datapath (Reg File) is the only part that is used by more than one stage of the pipeline. This is OK because the register file has independent Read and Write ports.
More specifically, the Reg/Decode stage of the pipeline uses the register file’s read port while the Write Back stage of the pipeline uses the register file’s write port.
+3 = 32 min. (Y:12) PC
PC+4
Imm16
Imm16 Rs
Zero
busA
Data
Mem A Ra
busB
Exec
Unit Rb RA Do
IUnit
IF/ID Register
ID/Ex Register
Ex/Mem Register
Mem/Wr Register
Mux 1 Rt
RFile WA Di Rt Rw Di I Rd 1
RegDst
ALUSrc
MemWr
MemtoReg

3. Review: Pipeline Control “Data Stationary Control”
The Main Control generates the control signals during Reg/Dec
Control signals for Exec (ExtOp, ALUSrc, ...) are used 1 cycle later
Control signals for Mem (MemWr Branch) are used 2 cycles later
Control signals for Wr (MemtoReg MemWr) are used 3 cycles later
Reg/Dec
Exec
Mem Wr
ExtOp
ExtOp
The main control here is identical to the one in the single cycle processor. It generate all the control signals necessary for a given instruction during that instruction’s Reg/Decode stage.
All these control signals will be saved in the ID/Exec pipeline register at the end of the Reg/Decode cycle.
The control signals for the Exec stage (ALUSrc, ... etc.) come from the output of the ID/Exec register. That is they are delayed ONE cycle from the cycle they are generated.
The rest of the control signals that are not used during the Exec stage is passed down the pipeline and saved in the Exec/Mem register.
The control signals for the Mem stage (MemWr, Branch) come from the output of the Exec/Mem register. That is they are delayed two cycles from the cycle they are generated.
Finally, the control signals for the Wr stage (MemtoReg & RegWr) come from the output of the Exec/Wr register: they are delayed three cycles from the cycle they are generated.
+2 = 45 min. (Y:45)
ALUSrc
ALUSrc
ALUOp
ALUOp
Main
Control
RegDst
RegDst
Ex/Mem Register
IF/ID Register
ID/Ex Register
Mem/Wr Register
MemWr
MemWr
MemWr
Branch
Branch
Branch
MemtoReg
MemtoReg
MemtoReg
MemtoReg
RegWr
RegWr
RegWr
RegWr

4. Review: Pipeline Summary
Pipeline Processor:
Natural enhancement of the multiple clock cycle processor
Each functional unit can only be used once per instruction
If a instruction is going to use a functional unit:
it must use it at the same stage as all other instructions
Pipeline Control:
Each stage’s control signal depends ONLY on the instruction that is currently in that stage
Let me summarized the different processor implementations we have learned in the last few lectures.
First the single cycle processor has two disadvantages: (1) long cycle time. (2) And may be more importantly, this long cycle time is too long for all instructions except the load.
The multiple clock cycle processor solve these problems by:
(1) Dividing the instructions into smaller steps.
(2) Then execute each step (instead of the entire instruction) in one clock cycle.
The pipeline processor is just a natural extension of the multiple clock cycle processor.
In order to convert the multiple clock cycle processor into a pipeline processor, we need to make sure each functional unit is only used once per instruction.
Furthermore, if a instruction is going to use a functional unit, it must use it at the SAME stage as all other instructions use it.
Finally, for the pipeline control, the most important thing you need to remember is that the control signals at each stage depends ONLY on the instruction that is currently in that stage.
+2 = 75 min. (Y:55)

5. Outline of Today’s Lecture
Recap and Introduction
Introduction to Hazards
Forwarding
1 cycle Load Delay
1 cycle Branch Delay
What makes pipelining hard
Summary
Here is the outline of today’s lecture.
In the next 15 minutes or so, I will give you an overview of the pipelining idea.
Then I will show you the pipelined datapath as well as the pipelined control.
One problem will come up again is the potential race condition between the address lines and the write enable line of data memory and register file. I will show you how to avoid that.
I will also show you how the pipelined datapath works with multiple instructions in it.
Finally, I will summarize the differences between the single cycle, multiple cycle and pipeline implementations of the processor.
+1 = 6 min. (X:46)

6. Its not that easy for computers
Limits to pipelining: Hazards prevent next instruction from executing during its designated clock cycle
structural hazards: HW cannot support this combination of instructions
data hazards: instruction depends on result of prior instruction still in the pipeline
control hazards: pipelining of branches & other instructions that change the PC
Common solution is to stall the pipeline until the hazard is resolved, inserting one or more “bubbles” in the pipeline

7. Single Memory is a Structural Hazard
Time (clock cycles)
ALU I n s t r. O r d e
Mem
Reg
Mem
Reg
Load
ALU
Mem
Reg
Instr 1
ALU
Mem
Reg
Instr 2
ALU
Instr 3
Mem
Reg
Mem
Reg
ALU
Mem
Reg
Instr 4

8. Option 1: Stall to resolve Memory Structural Hazard
Time (clock cycles)
ALU I n s t r. O r d e
Mem
Reg
Mem
Reg
Load
ALU
Mem
Reg
Instr 1
ALU
Mem
Reg
Instr 2
Instr 3(stall)
ALU
Mem
Reg
bubble
Instr 4
ALU
Mem
Reg

9. Option 2: Duplicate to Resolve Structural Hazard
Separate Instruction Cache (Im) & Data Cache (Dm)
Time (clock cycles)
ALU Im
Reg Dm I n s t r. O r d e
Load
ALU Im
Reg Dm
Instr 1
ALU Im
Reg Dm
Instr 2 Im
ALU
Reg Dm
Instr 3
ALU Im
Reg Dm
Instr 4

10. add r1 ,r2,r3 sub r4, r1 ,r3 and r6, r1 ,r7 or r8, r1 ,r9
Data Hazard on r1
add r1 ,r2,r3
sub r4, r1 ,r3
and r6, r1 ,r7
or r8, r1 ,r9
xor r10, r1 ,r11

11. Data Hazard on r1: (Figure 6.30, page 397, P&H)
Dependencies backwards in time are hazards
Time (clock cycles) IF
ID/RF EX
MEM WB
add r1,r2,r3 Im
Reg
ALU
Reg Dm I n s t r. O r d e
ALU
sub r4,r1,r3 Im
Reg Dm
Reg
ALU
and r6,r1,r7 Im
Reg Dm
Reg
or r8,r1,r9 Im
Reg
ALU Dm
Reg
ALU Im
Reg Dm
xor r10,r1,r11

12. Option1: HW Stalls to Resolve Data Hazard
Dependencies backwards in time are hazards
Time (clock cycles) IF
ID/RF EX
MEM WB
add r1,r2,r3 Im
Reg
ALU
Reg Dm I n s t r. O r d e
sub r4, r1,r3
bubble Im
Reg
ALU Dm
Reg
and r6,r1,r7
ALU Im
Reg Dm
or r8,r1,r9
ALU Im
Reg
xor r10,r1,r11 Im
Reg

13. But recall use of “Data Stationary Control”
The Main Control generates the control signals during Reg/Dec
Control signals for Exec (ExtOp, ALUSrc, ...) are used 1 cycle later
Control signals for Mem (MemWr Branch) are used 2 cycles later
Control signals for Wr (MemtoReg MemWr) are used 3 cycles later
Reg/Dec
Exec
Mem Wr
ExtOp
ExtOp
The main control here is identical to the one in the single cycle processor. It generate all the control signals necessary for a given instruction during that instruction’s Reg/Decode stage.
All these control signals will be saved in the ID/Exec pipeline register at the end of the Reg/Decode cycle.
The control signals for the Exec stage (ALUSrc, ... etc.) come from the output of the ID/Exec register. That is they are delayed ONE cycle from the cycle they are generated.
The rest of the control signals that are not used during the Exec stage is passed down the pipeline and saved in the Exec/Mem register.
The control signals for the Mem stage (MemWr, Branch) come from the output of the Exec/Mem register. That is they are delayed two cycles from the cycle they are generated.
Finally, the control signals for the Wr stage (MemtoReg & RegWr) come from the output of the Exec/Wr register: they are delayed three cycles from the cycle they are generated.
+2 = 45 min. (Y:45)
ALUSrc
ALUSrc
ALUOp
ALUOp
Main
Control
RegDst
RegDst
Ex/Mem Register
IF/ID Register
ID/Ex Register
Mem/Wr Register
MemWr
MemWr
MemWr
Branch
Branch
Branch
MemtoReg
MemtoReg
MemtoReg
MemtoReg
RegWr
RegWr
RegWr
RegWr

14. Option 1: How HW really stalls pipeline
HW doesn’t change PC => keeps fetching same instruction & sets control signals to benign values (0)
Time (clock cycles) IF
ID/RF EX
MEM WB
add r1,r2,r3 Im
Reg
ALU
Reg Dm I n s t r. O r d e
bubble Im
stall
bubble Im
stall
bubble Im
stall
sub r4,r1,r3
ALU Im
Reg Dm
and r6,r1,r7
ALU Im
Reg Dm

15. Option 2: SW inserts indepdendent instructions
Worst case inserts NOP instructions
Time (clock cycles) IF
ID/RF EX
MEM WB
add r1,r2,r3 Im
Reg
ALU
Reg Dm I n s t r. O r d e
ALU
nop Im
Reg Dm
Reg
ALU Im Dm
Reg
nop
Reg Im
Reg
ALU Dm
Reg
nop
sub r4,r1,r3
ALU Im
Reg Dm
and r6,r1,r7
ALU Im
Reg Dm

16. Questions and Administrative Matters
Do you have any questions before we go any further.
Let’s take care of some business before we continue.
+5 = 25 min. (Y:05)

17. Option 3 Insight: Data is available! )
Pipeline registers already contain needed data
Time (clock cycles) IF
ID/RF EX
MEM WB
add r1,r2,r3 Im
Reg
ALU
Reg Dm I n s t r. O r d e
ALU
sub r4,r1,r3 Im
Reg Dm
Reg
ALU
and r6,r1,r7 Im
Reg Dm
Reg
or r8,r1,r9 Im
Reg
ALU Dm
Reg
ALU Im
Reg Dm
xor r10,r1,r11

18. HW Change for “Forwarding” (Bypassing):)
Increase multiplexors to add paths from pipeline registers
Assumes register read during write gets new value (otherwise more results to be forwarded)

19. From Last Lecture: The Delay Load Phenomenon
Cycle 1
Cycle 2
Cycle 3
Cycle 4
Cycle 5
Cycle 6
Cycle 7
Cycle 8
Clock
I0: Load
Ifetch
Reg/Dec
Exec
Mem Wr
Plus 1
Ifetch
Reg/Dec
Exec
Mem Wr
Plus 2
Ifetch
Reg/Dec
Exec
Mem Wr
Plus 3
Ifetch
Reg/Dec
Exec
Mem Wr
Plus 4
Ifetch
Reg/Dec
Exec
Mem Wr
Similarly, although the load instruction is fetched during cycle 1, the data is not written into the register file until the end of Cycle 5.
Consequently, the earliest time we can read this value from the register file is in Cycle 6.
In other words, there is a 3-instruction delay between the load instruction and the instruction that can use the result of the load.
This is referred to as Data Hazard in the text book.
We will show in the next lecture that by clever design techniques, we can reduce this delay to ONE instruction.
That is if the load instruction is issued in Cycle 1, the instruction comes right next to it (Plus 1) cannot use the result of this load but the next-next instruction (Plus 2) can.
+2 = 73 min. (Y:53)
Although Load is fetched during Cycle 1:
The data is NOT written into the Reg File until the end of Cycle 5
We cannot read this value from the Reg File until Cycle 6
3-instruction delay before the load take effect

20. Forwarding reduces Data Hazard to 1 cycle:
Time (clock cycles) IF
ID/RF EX
MEM WB
lw r1, 0(r2) Im
Reg
ALU
Reg Dm I n s t r. O r d e
ALU
sub r4,r1,r6 Im
Reg Dm
Reg
ALU
and r6,r1,r7 Im
Reg Dm
Reg
or r8,r1,r9 Im
Reg
ALU Dm
Reg

21. Option1: HW Stalls to Resolve Data Hazard
“Interlock”: checks for hazard & stalls
Time (clock cycles) IF
ID/RF EX
MEM WB
lw r1, 0(r2) Im
Reg
ALU
Reg Dm I n s t r. O r d e
bubble Im
stall
ALU
sub r4,r1,r3 Im
Reg Dm
Reg
ALU
and r6,r1,r7 Im
Reg Dm
Reg
or r8,r1,r9 Im
Reg
ALU Dm
Reg

22. Option 2: SW inserts independent instructions
Worst case inserts NOP instructions
MIPS I solution: No HW checking
Time (clock cycles) IF
ID/RF EX
MEM WB
ALU Im
Reg Dm
lw r1, 0(r2) I n s t r. O r d e
ALU Im
Reg Dm
nop
ALU
sub r4,r1,r3 Im
Reg Dm
Reg
ALU
and r6,r1,r7 Im
Reg Dm
Reg
or r8,r1,r9 Im
Reg
ALU Dm
Reg

23. Software Scheduling to Avoid Load Hazards
Try producing fast code for
a = b + c;
d = e – f;
assuming a, b, c, d ,e, and f
in memory.
Slow code:
LW Rb,b
LW Rc,c
ADD Ra,Rb,Rc
SW a,Ra
LW Re,e
LW Rf,f
SUB Rd,Re,Rf
SW d,Rd

24. Software Scheduling to Avoid Load Hazards
Try producing fast code for
a = b + c;
d = e – f;
assuming a, b, c, d ,e, and f
in memory.
Slow code:
LW Rb,b
LW Rc,c
ADD Ra,Rb,Rc
SW a,Ra
LW Re,e
LW Rf,f
SUB Rd,Re,Rf
SW d,Rd
Fast code:
LW Rb,b
LW Rc,c
LW Re,e
ADD Ra,Rb,Rc
LW Rf,f
SW a,Ra
SUB Rd,Re,Rf
SW d,Rd

25. Compiler Avoiding Load Stalls:

26. From Last Lecture: The Delay Branch Phenomenon
Cycle 4
Cycle 5
Cycle 6
Cycle 7
Cycle 8
Cycle 9
Cycle 10
Cycle 11
Clk
12: Beq
(target is 1000)
Ifetch
Reg/Dec
Exec
Mem Wr
Ifetch
Reg/Dec
Exec
Mem Wr
16: R-type
20: R-type
Ifetch
Reg/Dec
Exec
Mem Wr
24: R-type
Ifetch
Reg/Dec
Exec
Mem Wr
1000: Target of Br
Ifetch
Reg/Dec
Exec
Mem Wr
Notice that although the branch instruction is fetched during Cycle 4, its target address is NOT written into the Program Counter until the end of clock Cycle 7.
Consequently, the branch target instruction is not fetched until clock Cycle 8.
In other words, there is a 3-instruction delay between the branch instruction is issued and the branch effect is felt in the program.
This is referred to as Branch Hazard in the text book.
And as we will show in the next lecture, by clever design techniques, we can reduce the delay to ONE instruction.
That is if the branch instruction in Location 12 is issued in Cycle 4, we will only execute one more sequential instruction (Location 16) before the branch target is executed.
+2 = 71 min. (Y:51)
Although Beq is fetched during Cycle 4:
Target address is NOT written into the PC until the end of Cycle 7
Branch’s target is NOT fetched until Cycle 8
3-instruction delay before the branch take effect

27. Control Hazard on Branches: 3 stage stall

28. Branch Stall Impact
If CPI = 1, 30% branch, Stall 3 cycles => new CPI = 1.9!
2 part solution:
Determine branch taken or not sooner, AND
Compute taken branch address earlier
MIPS branch tests = 0 or ° 0
Solution Option 1:
Move Zero test to ID/RF stage
Adder to calculate new PC in ID/RF stage
1 clock cycle penalty for branch vs. 3

29. Option 1: move HW forward to reduce branch delay
Instruction
Fetch
Instr. Decode
Reg. Fetch
Execute
Addr. Calc.
Memory
Access
Write
Back

30. Branch Delay now 1 clock cycle
Instruction
Fetch
Instr. Decode
Reg. Fetch
Execute
Addr. Calc.
Memory
Access
Write
Back

31. Option 2: Define Branch as Delayed
Worst case, SW inserts NOP into branch delay
Where get instructions to fill branch delay slot?
Before branch instruction
From the target address: only valuable when branch
From fall through: only valuable when don’t branch
Compiler effectiveness for single branch delay slot:
Fills about 60% of branch delay slots
About 80% of instructions executed in branch delay slots useful in computation
about 50% (60% x 80%) of slots usefully filled

32. When is pipelining hard?
Interrupts: 5 instructions executing in 5 stage pipeline
How to stop the pipeline?
Restrart?
Who caused the interrupt?
Stage Problem interrupts occurring
IF Page fault on instruction fetch; misaligned memory access; memory-protection violation
ID Undefined or illegal opcode
EX Arithmetic interrupt
MEM Page fault on data fetch; misaligned memory access; memory-protection violation

33. When is pipelining hard?
Complex Addressing Modes and Instructions
Address modes: Autoincrement causes register change during instruction execution
Interrupts?
Now worry about write hazards since write no longer last stage
Write After Read (WAR): Write occurs before independent read
Write After Write (WAW): Writes occur in wrong order, leaving wrong result in registers
(Previous data hazard called RAW, for Read After Write)
Memory-memory Move instructions
Multiple page faults
make progress?

34. When is pipelining hard?
Floating Point: long execution time
Also, may pipeline FP execution unit so that can initiate new instructions without waiting full latency
FP Instruction Latency Initiation Rate (MIPS R4000)
Add, Subtract 4 3
Multiply 8 4
Divide 36 35
Square root
Negate 2 1
Absolute value 2 1
FP compare 3 2
Divide, Square Root take ­10X to ­30X longer than Add
Exceptions?
Adds WAR and WAW hazards since pipelines are no longer same length

35. Hazard Detection
Suppose instruction i is about to be issued and a predecessor instruction j is in the instruction pipeline.
Rregs ( i ) = Registers read by instruction i
Wregs ( i ) = Registers written by instruction i
° A RAW hazard exists on register r if $ r, r Î Rregs( i ) Ç Wregs( j )
– Keep a record of pending writes (for inst's in the pipe) and compare with operand regs of current instruction.
– When instruction issues, reserve its result register.
– When on operation completes, remove its write reservation.
° A WAW hazard exists on register r if $ r, r Î Wregs( i ) Ç Wregs( j )
° A WAR hazard exists on register r if $ r, r Î Wregs( i ) Ç Rregs( j )

36. Avoiding Data Hazards by Design
Suppose instructions are executed in a pipelined fashion such that Instructions are initiated in order.
° WAW avoidance: if writes to a particular resource (e.g., reg) are performed in the same stage for all instructions, then no WAW hazards occur.
proof: writes are in the same time sequence as instructions.
° WAR avoidance: if in all instructions reads of a resource occur at an earlier stage than writes to that resource occur in any instruction, then no WAR hazards occur.
proof: A successor instruction must issue later, hence it will perform writes only after all reads for the current instruction.
I R/D E W
I R/D E W
I R/D E W

37. First Generation RISC Pipelines
° All instructions follow same pipeline order (“static schedule”).
° Register write in last stage
– Avoid WAW hazards
° All register reads performed in first stage after issue.
– Avoid WAR hazards
° Memory access in stage 4
– Avoid all memory hazards
° Control hazards resolved by delayed branch (with fast path)
° RAW hazards resolved by bypass, except on load results
which are resolved by fiat (delayed load).
Substantial pipelining with very little cost or complexity.
Machine organization is (slightly) exposed!
Relies very heavily on "hit assumption"of memory accesses in cache

38. Review: Summary of Pipelining Basics
Speed Up Š Pipeline Depth; if ideal CPI is 1, then:
Hazards limit performance on computers:
structural: need more HW resources
data: need forwarding, compiler scheduling
control: early evaluation & PC, delayed branch, prediction
Increasing length of pipe increases impact of hazards since pipelining helps instruction bandwidth, not latency
Compilers key to reducing cost of data and control hazards
load delay slots
branch delay slots
Exceptions, Instruction Set, FP makes pipelining harder
Longer pipelines => Branch prediction, more instruction parallelism?