1. 1 ECE462/562 ISA and Datapath Review Ali Akoglu

2. 2 Instruction Set Architecture A very important abstraction –interface between hardware and low-level software –standardizes instructions, machine language bit patterns, etc. –advantage: different implementations of the same architecture Modern instruction set architectures: –IA-32, PowerPC, MIPS, SPARC, ARM, and others

3. 3 MIPS arithmetic All instructions have 3 operands Operand order is fixed (destination first) Example: C code: a = b + c MIPS ‘code’: add a, b, c

4. 4 MIPS arithmetic Design Principle: simplicity favors regularity. Of course this complicates some things... C code: a = b + c + d; MIPS code: add a, b, c add a, a, d Operands must be registers, only 32 registers provided Each register contains 32 bits

5. 5 Registers vs. Memory ProcessorI/O Control Datapath Memory Input Output Arithmetic instructions operands must be registers, — only 32 registers provided Compiler associates variables with registers What about programs with lots of variables

6. 6 Memory Organization Viewed as a large, single-dimension array, with an address. A memory address is an index into the array "Byte addressing" means that the index points to a byte of memory. 0 1 2 3 4 5 6... 8 bits of data

7. 7 Memory Organization Bytes are nice, but most data items use larger "words" For MIPS, a word is 32 bits or 4 bytes. 2 32 bytes with byte addresses from 0 to 2 32 -1 2 30 words with byte addresses 0, 4, 8,... 2 32 -4 Words are aligned i.e., what are the least 2 significant bits of a word address? 0 4 8 12... 32 bits of data Registers hold 32 bits of data

8. 8 So far we’ve learned: MIPS — loading words but addressing bytes — arithmetic on registers only InstructionMeaning add $s1, $s2, $s3$s1 = $s2 + $s3 sub $s1, $s2, $s3$s1 = $s2 – $s3 lw $s1, 100($s2)$s1 = Memory[$s2+100] sw $s1, 100($s2)Memory[$s2+100] = $s1

9. 9 Instructions Load and store instructions Example: C code: A[12] = h + A[8]; # $s3 stores base address of A and $s2 stores h MIPS code: lw $t0, 32($s3) add $t0, $s2, $t0 sw $t0, 48($s3) Remember arithmetic operands are registers, not memory! Can’t write: add 48($s3), $s2, 32($s3)

10. 10 Summary A[300]=h+A[300] # $t1 = base address of A, $s2 stores h # use $t0 for temporary register Lw $t0,1200($t1) Add $t0, $s2, $t0 Sw $t0, 1200($t1) Op rs,rt,address 35,9,8,1200 Op,rs,rt,rd,shamt,funct 0,18,8,8,0,32 Op,rs,rt,address 43,9,8,1200

11. 11 Policy of Use Conventions Register 1 ($at) reserved for assembler, 26-27 for operating system

12. 12 MIPS Format

13. 13 Consider the load-word and store-word instructions Introduce a new type of instruction format –I-type for data transfer instructions –other format was R-type for register Example: lw $t0, 32($s2) 35 18 8 32 op rs rt 16 bit number Machine Language

14. 14 Shift and Logical Operations

15. 15 Summary of New Instructions

16. 16 Control Instructions

17. 17 Instructions: bne $t4,$t5,Label Next instruction is at Label if $t4≠$t5 beq $t4,$t5,Label Next instruction is at Label if $t4=$t5 Formats: op rs rt 16 bit address I Addresses in Branches

18. 18 Instructions: bne $t4,$t5,Label if $t4 != $t5 beq $t4,$t5,Label if $t4 = $t5 j Label j Label Next instruction is at Label Formats: op rs rt 16 bit address op 26 bit address IJIJ Addresses in Branches and Jumps

19. 19 simple instructions all 32 bits wide very structured, no unnecessary baggage only three instruction formats op rs rt rdshamtfunct op rs rt 16 bit address op 26 bit address RIJRIJ Overview of MIPS

20. 20 Datapath add $t1, $s1, $s2 ($t1=9, $s1=17, $s2=18) –00000010001100100100100000100000 op rs rt rdshamtfunct

21. 21 MIPS64 - Instruction Set Architecture (ISA) MIPS is a compact RISC architecture (register-to-register) Simple 64-bit (register) Load/Store architecture (data – 64 bits, instructions – 32 bits) Design for pipelining efficiency Components – Instructions (Types, Syntax) Registers (number, function) Addressing modes for MIPS Data Transfers Data Types (double, float…)

22. 22 MIPS64 Registers 32 general-purpose registers (64-bits) –R0, R1, … R31 –works for any instructions that involves integers, including memory. 32 floating-point registers –F0, F1…., F30, F31 –For single precision (32-bits) (other 32 bits unused) –For double precision (64-bits)

23. 23 MIPS64 Instruction Set Examples Arithmetic/Logical –Add unsigned DADDU R1, R2, R3 -- Regs[R1] Regs[R2] + Regs[R3] –Shift Left Logical DSLL R1, R2, #5 -- Regs[R1] Regs[R2] << 5 Load/Store –Load Double word LD R1, 30(R2) -- Regs[R1] 64 Mem[30+Regs[R2]] –Store FP single S.S F0, 40(R3) -- Mem[40+Regs[R3]] 32 Regs[F0] 0…31 Control –Jump Register JR R3 -- PC Regs[R3] –Branch Not Equal BNE R3, R4, name -- if (Regs[R3] != Regs[R4]), PC name; ((PC+4) – 2 17 ) ≤ name < ((PC+4) + 2 17 )

24. 24 MIPS64 Instruction Set Architecture For more, please refer to – –Appendix A of the book (5 th Edition)

25. 25 The simple datapath

26. 26

27. 27 Datapath in Operation for R-Type Instruction

28. 28 Datapath in Operation for Load Instruction

29. 29 Datapath in Operation for Branch Equal Instruction

30. 30 Single Cycle Problems –Wasteful of area Each unit used once per clock cycle –Clock cycle equal to worst case scenario Will reducing the delay of common case help?

31. 31 Pipelining: It’s Natural! Ann, Brian, Cathy, Dave each have one load of clothes to wash, dry, and fold Washer takes 30 minutes Dryer takes 40 minutes “Folder” takes 20 minutes ABCD

32. 32 Sequential Laundry Sequential laundry takes 6 hours for 4 loads If they learned pipelining, how long would laundry take? ABCD 304020304020304020304020 6 PM 789 10 11 Midnight TaskOrderTaskOrder Time

33. 33 Pipelined Laundry: Start work ASAP Pipelined laundry takes 3.5 hours for 4 loads ABCD 6 PM 789 10 11 Midnight TaskOrderTaskOrder Time 3040 20

34. 34 Pipelining Lessons Pipelining doesn’t help latency of single task, it helps throughput of entire workload Pipeline rate limited by slowest pipeline stage ABCD 6 PM 78 9 TaskOrderTaskOrder Time 3040 20 What is the speedup of a pipeline of n stages?

35. 35 Pipelining Improve performance by increasing instruction throughput Ideal speedup is number of stages in the pipeline. Do we achieve this?

36. 36 Basic Idea What do we need to add to actually split the datapath into stages?

37. 37 Pipelined datapath

38. 38 Five Stages (lw) Memory and registers Left half: write Right half: read

39. 39 Five Stages (lw)

40. 40 Five Stages (lw)

41. 41 What is wrong with this datapath?

42. 42 Store Instruction

43. 43 Store Instruction

44. 44 Pipeline control

45. 45 Pipeline control

46. 46 Datapath with control

47. 47 Pipelining is not quite that easy! Limits to pipelining: Hazards prevent next instruction from executing during its designated clock cycle –Structural hazards: HW cannot support this combination of instructions (single person to fold and put clothes away) –Data hazards: Instruction depends on result of prior instruction still in the pipeline (missing sock) –Control hazards: Caused by delay between the fetching of instructions and decisions about changes in control flow (branches and jumps).

48. 48 One Memory Port/Structural Hazards Figure A.4, Page A-14

49. 49 Inst I before inst j in in the program Read After Write (RAW) Instr J tries to read operand before Instr I writes it Caused by a “Dependence” (in compiler nomenclature). This hazard results from an actual need for communication. Three Generic Data Hazards I: add r1,r2,r3 J: sub r4,r1,r3

50. 50 Write After Read (WAR) Instr J writes operand before Instr I reads it Called an “anti-dependence” by compiler writers. This results from reuse of the name “r1”. Can’t happen in MIPS 5 stage pipeline because: I: sub r4,r1,r3 J: add r1,r2,r3 K: mul r6,r1,r7 Three Generic Data Hazards All instructions take 5 stages, and Reads are always in stage 2, and Writes are always in stage 5

51. 51 Three Generic Data Hazards Write After Write (WAW) Instr J writes operand before Instr I writes it. Called an “output dependence” by compiler writers This also results from the reuse of name “r1”. Can’t happen in MIPS 5 stage pipeline because: – All instructions take 5 stages, and – Writes are always in stage 5 I: sub r1,r4,r3 J: add r1,r2,r3 K: mul r6,r1,r7

52. 52 Representation

53. 53 Problem with starting next instruction before first is finished –Dependencies that “go backward in time” are data hazards Dependencies

54. 54 Hazards

55. 55 Use temporary results, don’t wait for them to be written –register file forwarding to handle read/write to same register –ALU forwarding Forwarding

56. 56 Forwarding sub$2, $1, $3 and$12, $2, $5 or$13, $6, $2 add$14, $2, $2 sw$15, 100($2)

57. 57 Forwarding Forward from EX/MEM registers If (EX/MEM.RegWrite) and If (EX/MEM.Rd != 0) and (ID/EX.Rs == EX/MEM.Rd) Forward from MEM/WB registers If (MEM/WB.RegWrite) and If (MEM/WB.Rd != 0) and If (ID/EX.Rt==EX/MEM.Rd)

58. 58 Can't always forward

59. 59 Can't always forward

60. 60 Load word can still cause a hazard: –an instruction tries to read a register following a load instruction that writes to the same register. Can't always forward

61. 61 Stalling Hardware detection and no-op insertion is called stalling Stall pipeline by keeping instruction in the same stage

62. 62

63. 63 Pipeline with hazard detection

64. 64 Assume that register file is written in the first half and read in the second half of the clock cycle. Cycles 1 2 3 4 5 6 7 8 9 10 11 12 13 load r2 <- mem(r1+0) r3 <- r3 + r2 load r4 <- mem(r2+r3) r4 <- r5 - r3 IFIDEXMEWB IFIDSSEXMEWB IFSSIDEXMEWB IFIDSEXMEWBSS load r2 <- mem(r1+0) ; LOAD1 r3 <- r3 + r2 ; ADD load r4 <- mem(r2+r3) ; LOAD2 r4 <- r5 - r3 ; SUB

65. 65 Summary

66. 66 Forwarding Case Summary

67. 67 Multi-cycle

68. 68 Multi-cycle

69. 69 Multi-cycle Pipeline

70. 70 Branch Hazards

71. 71 When we decide to branch, other instructions are in the pipeline! We are predicting “branch not taken” –need to add hardware for flushing instructions if we are wrong Branch hazards

72. 72 Branch detection in ID stage

73. 73 Solution to control hazards Branch prediction –We are predicting “branch not taken” –Need to add hardware for flushing instructions if we are wrong Reduce branch penalty –By advancing the branch decision to ID stage –Compare the data read from two registers read in ID stage –Comparison for equality is a simpler design! (Why?) –Still need to flush instruction in IF stage Make the hazard into a feature! –Delayed branch slot - Always execute instruction following branch

74. 74 Branch Prediction Sophisticated Techniques: –A “branch target buffer” to help us look up the destination –Correlating predictors that base prediction on global behavior and recently executed branches (e.g., prediction for a specific branch instruction based on what happened in previous branches) –Tournament predictors that use different types of prediction strategies and keep track of which one is performing best. –A “branch delay slot” which the compiler tries to fill with a useful instruction (make the one cycle delay part of the ISA) Branch prediction is especially important because it enables other more advanced pipelining techniques to be effective! Modern processors predict correctly 95% of the time!

75. 75 Four Branch Hazard Alternatives #1: Stall until branch direction is clear: branch penalty is fixed and can not be reduced by software (this is the example of MIPS) #2: Predict Branch Not Taken (treat every branch as not taken) –Execute successor instructions in sequence –“flush” instructions in pipeline if branch actually taken –47% MIPS branches not taken on average –PC+4 already calculated, so use it to get next instruction

76. 76 Four Branch Hazard Alternatives: # 3: Predict Branch Taken (treat every branch as taken) As soon as the branch is decoded and the target address is computed, we assume the branch is taken and begin fetching and executing at the target address. –53% MIPS branches taken on average –Because in our MIPS pipeline we don’t know the target address any earlier than we know the branch outcome, there is no advantage in this approach for MIPS. –MIPS still incurs 1 cycle branch penalty Other machines: branch target known before outcome

77. 77 Four Branch Hazard Alternatives #4: Delayed Branch –In a delayed branch, the execution cycle with a branch delay of length n is: branch instruction sequential successor 1 sequential successor 2........ sequential successor n branch target if taken These sequential successor instructions are in a branch-delay slots. The sequential successors are executed whether or not the branch is taken. Branch delay of length n The job of the compiler is to make the successor instructions valid and useful.

78. 78 Scheduling Branch Delay Slots (Fig A.14) add $1,$2,$3 if $2=0 then delay slot A. From before branchB. From branch targetC. From fall through add $1,$2,$3 if $1=0 then delay slot add $1,$2,$3 if $1=0 then delay slot sub $4,$5,$6 becomes if $2=0 then add $1,$2,$3 if $1=0 then sub $4,$5,$6 add $1,$2,$3 if $1=0 then Or $7, $8, $9 Sub $4, $5, $6 Or $7, $8, $ 9 Sub $4,$5,$6

79. 79 Delayed Branch Where to get instructions to fill branch delay slot? –Before branch instruction: this is the best choice if feasible. –From the target address: only valuable when branch taken –From fall through: only valuable when branch not taken Compiler effectiveness for single branch delay slot: –Fills about 60% of branch delay slots –About 80% of instructions executed in branch delay slots useful in computation –About 50% (60% x 80%) of slots usefully filled Delayed Branch downside: As processor go to deeper pipelines and multiple issue, the branch delay grows and need more than one delay slot –Delayed branching has lost popularity compared to more expensive but more flexible dynamic approaches –Growth in available transistors has made dynamic approaches relatively cheaper

80. 80 Improving Performance Try and avoid stalls! E.g., reorder these instructions: lw $t0, 0($t1) lw $t2, 4($t1) sw $t2, 0($t1) sw $t0, 4($t1) Dynamic Pipeline Scheduling –Hardware chooses which instructions to execute next –Will execute instructions out of order (e.g., doesn’t wait for a dependency to be resolved, but rather keeps going!) –Speculates on branches and keeps the pipeline full (may need to rollback if prediction incorrect) Trying to exploit instruction-level parallelism

81. 81 Advanced Pipelining Increase the depth of the pipeline Start more than one instruction each cycle (multiple issue) Loop unrolling to expose more ILP (better scheduling) “Superscalar” processors –DEC Alpha 21264: 9 stage pipeline, 6 instruction issue All modern processors are superscalar and issue multiple instructions usually with some limitations (e.g., different “pipes”)

82. 82 Source: For ( i=1000; i>0; i=i-1 ) x[i] = x[i] + s; Direct translation: –Loop:LDF0, 0 (R1);R1 points x[1000] ADDDF4, F0, F2;F2 = scalar value SDF4, 0(R1) R2 last element DADDUIR1, R1, #-8 BNER1, R2, loop; ProducerConsumerLatency FP ALU opAnother FP ALU op3 FP ALU opStore double2 Load doubleFP ALU op1 Store double 0 Assume 1 cycle latency from unsigned integer arithmetic to dependent instruction

83. 83 Reducing stalls Pipeline Implementation: Loop:LDF0, 0 (R1) stall ADDDF4, F0, F2 stall stall SDF4, 0(R1) DADDUI R1, R1, #-8 stall BNER1, R2, loop stall –Loop:LD F0, 0 (R1) DADDUI R1, R1, #-8 ADDD F4, F0, F2 stall stall SD F4, 8(R1) BNE R1, R2, Loop ProducerConsumerLatency FP ALU opAnother FP ALU op3 FP ALU opStore double2 Load doubleFP ALU op1 Store double 0 123456789123456789

84. 84 LDF0, 0(R1) ADDDF4, F0, F2 SDF4, 0(R1); drop SUBI & BNEZ LDF6, -8 (R1) ADDDF8, F6, F2 SDF8, -8 (R1) ; drop SUBI & BNEZ LDF10, -16 (R1) ADDDF12, F10, F2 SDF12, -16 (R1) ; drop SUBI & BNEZ LDF14, -24 (R1) ADDDF16, F14, F2 SDF16, -24 (R1) DADDUIR1, R1, #-32 BNER1, R2, Loop Loop Unrolling Loop ProducerConsumerLatency FP ALU opAnother FP ALU op3 FP ALU opStore double2 Load doubleFP ALU op1 Store double 0 27 cycles: 14 instructions, 1 for each LD, 2 for each ADDD, 1 for DADDUI

85. 85 LoopLDF0, 0(R1) LDF6, -8 (R1) LDF10, -16(R1) LDF14, -24(R1) ADDDF4, F0, F2 ADDDF8, F6, F2 ADDDF12, F10, F2 ADDDF16, F14, F2 SDF4, 0(R1) SDF8, -8 (R1) DADDUIR1, R1, #-32 SDF12, -16 (R1) SDF16, 8(R1) BNER1, R2, Loop 14 instructions (3.5 cycles per element vs. 9 cycles!) Design Issues: Code size! Instruction cache Register space Iteration dependence Loop termination Memory addressing

86. 86 Superscalar architecture -- Two instructions executed in parallel Loop unrolling? Branch delay slot?

87. 87 Dynamically scheduled pipeline

88. 88 Important facts to remember Pipelined processors divide execution in multiple steps However pipeline hazards reduce performance –Structural, data, and control hazard Data forwarding helps resolve data hazards –But all hazards cannot be resolved –Some data hazards require bubble or noop insertion Effects of control hazard reduced by branch prediction –Predict always taken, delayed slots, branch prediction table –Structural hazards are resolved by duplicating resources Time to execute n instructions depends on –# of stages (k) –# of control hazard and penalty of each step –# of data hazards and penalty for each –Time = n + k - 1 + (load hazard penalty) + (branch penalty) Load hazard penalty is 1 or 0 cycle –Depending on data use with forwarding Branch penalty is 3, 2, 1, or zero cycles depending on scheme

89. 89 Design and performance issues with pipelining Pipelined processors are not EASY to design Technology affect implementation Instruction set design affect the performance –i.e., beq, bne More stages do not lead to higher performance!