1. Out of Order (OoO) Execution
EE457
Out of Order (OoO) Execution
Introduction to Dynamic Scheduling of Instructions (The Tomasulo Algorithm) By
Gandhi Puvvada

2. References EE557 Textbook Prof. Dubois’ EE557 Classnotes
Prof. Annavaram’s slides
Prof. Patterson’s Lecture slides

3. Programs often have several small fragments of code, which can be executed in any order.

4. OoO (Out of Order) execution
Io = In order ”Execution” here means producing the results. Completion means committing results. (writing into register file or memory). IoI (IoD)  OoE  IoC In order Issue/Dispatch, Out of order Execution and finally In order completion/commitment

5. IoC or OoC?
IoI (IoD)  OoE  IoC IoC (In order completion) is necessary to support exceptions (ex: page fault). Here we present first IoI (IoD)  OoE  OoC and then (at the end) IoI (IoD)  OoE  IoC

6. OoC? But branches ..
OoC? Hope you are not executing instruction beyond a branch and committing them! Well we dispatch a branch and suspend dispatching and wait until the branch is resolved. Then we resume dispatching instructions beyond the branch at either the fall-through area or at the target area.

7. Instruction Scheduling (Re-ordering of instructions)
Basic block = a straight-line code sequence with no branches.
Compiler can perform static instruction scheduling.
Tomasulo Algorithm lets us schedule instructions dynamically (in hardware).
Branch prediction and speculative execution beyond a branch (of course with ability to flush wrong-path instructions on misprediction) will be covered later (and implemented on FPGA in EE560).

8. Register renaming to allow later instructions to proceed
lw $8, 40($2);
add $8, $8, $8;
sw $8, 40($2);
lw $8, 60($3);
sw $8, 60($3);
lw $8, 40($2);
add $8, $8, $8;
sw $8, 40($2);
lw $48, 60($3);
add $48, $48, $48;
sw $48, 60($3);

9. Static Scheduling (based on Prof. Dubois slide)
Strengths -- Hardware simplicity -- Compiler has a global view of the code (does not help the hardware much)
Weaknesses -- can not be CPU-implementation specific -- can not foresee dynamic events -- cache misses -- data-dependent delays -- conditional branches  can only reschedule instructions in a basic block (basic block = a straight-line code sequence with no branches) -- can not pre-compute memory addresses

11. Simple 5-stage pipeline In-order execution RAW dependency Solve it by forwarding, if not, by stalling
Dependent instructions are stalled in the ID stage IM DM IF ID EX M WB

12. Simple 5-stage pipeline: Dependent instructions are stalled in the ID stage
and lw

13. Simple 5-stage pipeline: Dependent instructions can not be stalled in the EX stage. Why?
and lw

14. Provide multiple functional units (for simplicity, we avoid talking about floating point execution unit and floating point register file) Stall, after decoding, in queues
Divide
Multiply IM
Integer DM
Load/ Store IF ID WB
Queues and Functional unit

15. rs, rt (IDs) are carried into EX
Why junior instructions carry their source register IDs into EX stage? Well they need to get help from Senior #1 or Senior #2 in EX stage under the control of the FU. No more of that. There may be 40 seniors in front of you. So I, the dispatch unit, will tell you from which senior you need to get help for which source register.
rs, rt (IDs) are carried into EX

16. Tomasulo’s plan OoO Out of order execution
Multiple functional units (say, Integer, DM, Multiplier, Divider)
Queues between ID and EX stages (in place of ID/EX register)

17. Out of order execution ?! Problems all over ??!!
For the time, no branch prediction, no speculative execution beyond branches, just stall on a conditional branch
No support for precise exceptions for the time
Even then, …

18. RAW, WAR, and WAW
RAW = Read After Write lw $8, 40($2); add $9, $8, $7; WAR = Write after Read add $9, $8, $6; lw $8, 40($2); WAW = Write after Write add $9, $8, $6; lw $9, 40($2);
WAW ? How is it possible? Consider a printer or a FIFO
Why would anyone produce some result in $9 and without utilizing that result, why would he overwrite it with another result?

19. WAW can easily occur!
WAW ? How is it possible? In out of order execution, instructions before the branch and instruction after the branch can co-exist. For example, multiple iterations of this loop can coexist in the execution area. So, what?
Loop: LW $2, 40($1);
MULT $4 $2, $3;
SW $4, 40($1);
ADDI $1, $1, -4;
BNE $1, $0, Loop;

20. Say a company gives standard bonus to most of the employees and a higher bonus to the managers. So you load into $3 standard bonus from the stdbonus location in memory. And then you check to see if it is a case of a manager, and then load into $3 again (overwriting the earlier $3) the special bonus from the special location in memory.
LW $3 stdbonus ($0)
BNE $1, $2, SKIP
LW $3 special ($0)

21. RAW, WAR, and WAW (some terminology to remember)
RAW = Read After Write lw $8, 40($2); add $9, $8, $7; WAR = Write after Read add $9, $8, $6; lw $8, 40($2); WAW = Write after Write add $9, $8, $6; lw $9, 40($2);
RAW A true dependency
WAR An anti-dependency
Name Dependences
WAW An output dependency

22. RAW, WAR, and WAW In-order execution: We need to deal with RAW only.
Out of order execution: Now we need to deal with WAR and WAW besides RAW.

24. Limited Architectural Registers More Physical Registers Register Renaming
lw $8, 40($2);
add $8, $8, $8;
sw $8, 40($2);
lw $8, 60($3);
sw $8, 60($3);
It is clear that compiler is using $8 as a temporary register. If there is a delay in obtaining $2, the first part of the code can not proceed. Unfortunately, the second part of the code can not proceed because of name dependency for $8.

25. This is an example of name dependency.
If we had 64 registers instead of 32 registers, then perhaps compiler might have used $48 instead of $8 and we could have executed the second part of the code before the first part!
lw $8, 40($2);
add $8, $8, $8;
sw $8, 40($2);
lw $48, 60($3);
add $48, $48, $48;
sw $48, 60($3);
This is an example of name dependency.

26. Four different temporary registers can be used here as shown: $8, $18, $28, and $48 (or called with coded names, LION, TIGER, CAT, and ANT).
lw $8, 40($2);
add $18, $8, $8;
sw $18, 40($2);
lw $28, 60($3);
add $48, $28, $28;
sw $48, 60($3);
lw LION, 40($2);
add TIGER, LION, LION;
sw TIGER, 40($2);
lw CAT, 60($3);
add ANT, CAT, CAT;
sw ANT, 60($3);

27. Can a later implementation provide 64 registers (instead of 32) while maintaining binary compatibility with previously compiled codes?
Answer: Yes / No
Why?

28. Answer: Can not change the number of Architectural Registers Register Renaming Through Tagging Registers This solves name dependency problems (WAR and WAW) while attending to true dependency (RAW) through waiting in queues.

29. RST = Register Status Table RF = Register File
square_root $2, $10;
$1 $2 $3
$4 $5 $6 $7 $8 . .
. $31
$1 $2 $3
$4 $5 $6 $7 $8 . .
. $31
lw $8, 40($2);
add $8, $8, $8;
sw $8, 40($2);
lw $8, 60($3);
sw $8, 60($3);
dependent source
destination
RST = Register Status Table RF = Register File

30. lw $8, 40($2); add $8, $8, $8; sw $8, 40($2); lw $8, 60($3);
RST RF
square_root $2, $10;
$1 $2 $3
$4 $5 $6 $7 $8 . .
. $31
$1 $2 $3
$4 $5 $6 $7 $8 . .
. $31
lw $8, 40($2);
add $8, $8, $8;
sw $8, 40($2);
lw $8, 60($3);
sw $8, 60($3);

31. lw $8, 40($2); add $8, $8, $8; sw $8, 40($2); lw $8, 60($3);
RST RF
square_root $2, $10;
$1 $2 $3
$4 $5 $6 $7 $8 . .
. $31
$1 $2 $3
$4 $5 $6 $7 $8 . .
. $31
lw $8, 40($2);
add $8, $8, $8;
sw $8, 40($2);
lw $8, 60($3);
sw $8, 60($3);

32. lw $8, 40($2); add $8, $8, $8; sw $8, 40($2); lw $8, 60($3);
RST RF
square_root $2, $10;
$1 $2 $3
$4 $5 $6 $7 $8 . .
. $31
$1 $2 $3
$4 $5 $6 $7 $8 . .
. $31
lw $8, 40($2);
add $8, $8, $8;
sw $8, 40($2);
lw $8, 60($3);
sw $8, 60($3);

33. lw $8, 40($2); add $8, $8, $8; sw $8, 40($2); lw $8, 60($3);
square_root $2, $10;
Dispatch unit decodes and dispatches instructions. For destination operand, an instruction carries a TAG (but not the actual register name)! For source operands, an instruction carries either the values or TAGs of the operands (but not the actual register names)!
lw $8, 40($2);
add $8, $8, $8;
sw $8, 40($2);
lw $8, 60($3);
sw $8, 60($3);

34. Register Renaming

35. TAGs for destinations or sources or for both?
A new tag is assigned to the destination register of the instruction being dispatched.
For each of the source registers (source operands) of the instruction being dispatched, either the value of the source register (if it has not been previously tagged) or the existing tag associated with the source register (if it has been tagged already) is conveyed to the instruction.
If a tag is conveyed for a source, then the instruction needs to wait for the original instruction with that destination tag to go on to the CDB and announce the value.

36. Unique TAG 4 4 Like SSN, we need a unique TAG SSNs are reused.
Similarly TAGs can be reused.
TAGs are similar to the number TOKENs.

37. Take a number vs. Take a token4
In State Bank of India, the cashier issues brass tokens to customers trying to draw money as an identification (and not at all to put them in any virtual queue). Token numbers are in random order. The cashier verifies the signature in the records room and returns with money, call the token number and issues the money. Tokens are reclaimed and reused.
Helps to create a Virtual Queue. We do not need that here!

38. TAGs (= Tokens) 4
How many Tokens should the bank cashier have to start with?
What happens if the tokens are run out?
Does he need to have any order in holding tokens and issuing tokens?
Does he have to collect tokens back?

39. TAG FIFO (FIFOs are taught in EE560)
To issue and collect Tokens (TAGs), use a circular FIFO (First-in-First-Out) unit. While the FIFO-order is not important here, a FIFO is the easiest to implement in hardware compared to a random order in a pile.
Filled with (say) 64 tokens (in any order) initially on reset.
Tokens return in out of order anyway.
Put tokens back in the FIFO and issue. 1 63 wp rp 2
Full wp 63 2 1 wp rp 2 rp 63
2 tokens issued
1 token returned

40. CDB = Common Data Bus (compare it to a Public Announcing System)
Block Diagram provided by Prof. Dubois
Simplified for EE457
TAG FIFO 63 2
Int. Divider
Integer Multiplier
Issue Unit
CDB = Common Data Bus (compare it to a Public Announcing System)

41. Front-End & Back-End IFQ Instruction Fetch Queue (a FIFO structure)
Dispatch unit (including RST, RF, Tag FIFO)
Load Store and other Issue Queues
Issue Unit
Functional units
CDB (Common Data Bus)

43. Bottle neck in the design
CDB = Common Data Bus
Do all instructions use CDB?
sw ?
j (jump)?
beq

44. load store queue Address calculation Memory disambiguation
Mr. Bruin: Let me take a guess! You will now propose to have a MST (Memory Status Table) (like the RST). And you will rename memory locations to solve WAW and WAR problems among memory locations, right?!

45. MST (Memory Status Table). No way. It is too big
MST (Memory Status Table)? No way! It is too big! We will just ask the junior to stall and wait to solve his WAR and WAW problems with his seniors.
$1 $2 $3
$4 $5 $6 $7 $8 . .
. $31
RST RF .
0 1
. .
MST
Memory

46. Address calculation for lw and sw
EE557 approach for address calculation
EE457/560 approach for address calculation
Dedicated adder, to compute address, attached to the load-store queue.

47. Memory Disambiguation
EE557

48. Memory Disambiguation
RAW sw $2, 2000($0);
lw $8, 2000($0);
WAW sw $2, 2000($0);
sw $8, 2000($0);
WAR lw $2, 2000($0);
sw $8, 2000($0);

49. Memory Disambiguation
RAW sw $2, 2000($0);
lw $8, 2000($0);
This later lw can proceed only if there is no store ahead of it with the same address.
WAW sw $2, 2000($0);
sw $8, 2000($0);
This later sw can proceed only if there is no store ahead of it with the same address.
WAR lw $2, 2000($0);
sw $8, 2000($0);
This later sw can proceed only if there is no load ahead of it with the same address.

50. Maintaining instructions in the order of arrival (issue order/program order) in a queue
Is it necessary or is it desirable? In the case of L-S Queue ? In the case of Integer and other queues (mult queue, div queue)?

51. Maintaining instructions in the order of arrival (issue order/program order) in a queue
Is it necessary or is it desirable? In the case of L-S Queue ? NECESSARY to enforce memory disambiguation rules In the case of Integer and other queues (mult queue, div queue)? DESIRABLE, so that an earlier instruction gets executed whenever possible, there by perhaps reducing too many instructions waiting on it.

52. Priority (based on the order of arrival) among instructions ready to execute
Is it necessary or is it desirable?
Local priority with in the queues
Global priority across the queues

53. Issue Unit CDB availability constraint
Pipelined functional unit vs. Multi-cycle functional unit
Conflict resolution Round-robin priority adequate?, well, …

54. Conditional branches
Dispatch unit stops dispatching until the branch is resolved.
CDB broadcasts the result of the branch
Dispatching continues there after either at the fall-through instruction or at target instruction.
Successful branch shall cause flushing of IFQ very much like jump.

55. Conditional branches
Since we stop dispatching instructions after a branch, does it mean that this branch is the last instruction to be executed in the back-end ?
Is it possible that the back-end holds simultaneously (a) some instructions dispatched before the branch and (b) some instructions issued after the branch was resolved?

56. Tomasulo Loop Example Loop: LW $2, 40($1); MULT $4 $2, $3;
SW $4, 40($1);
ADDI $1, $1, -4;
BNE $1, $0, Loop;
Assume Multiply takes 4 clocks
Assume first load takes 8 clocks (cache miss), second load takes 1 clock (hit)
Based on Prof. Annavaram’s lecture slide

57. How could Tomasulo overlap iterations of loops?
Loop: LW $2, 40($1);
MULT $4 $2, $3;
SW $4, 40($1);
ADDI $1, $1, -4;
BNE $1, $0, Loop;
The destination registers bear different TAGs in different iterations. These tags were given in place of the source operands to the dependent instructions following them.

58. Say, only two iterations. Let us unroll the two iterations.
Loop: LW $2, 40($1);
MULT $4 $2, $3;
SW $4, 40($1);
ADDI $1, $1, -4;
BNE $1, $0, Loop; Loop: LW $2, 40($1);
BNE $1, $0, Loop;
destination register
dependent source register(s)

59. Because, there is no reorder buffer
Because, there is no reorder buffer. Note: Your EE560 project will use a reorder buffer and much more!