1. Energy-Proportional Photonic Interconnects Nikos Hardavellas PARAG@N – Parallel Architecture Group Northwestern University Team: Y. Demir, P. Yan, S. Song, J. Kim, G. Memik

2. Technology Scaling Runs Out of Steam Transistor counts increase exponentially, but… © Hardavellas 2 Can no longer power the entire chip (voltage, cooling do not scale)

3. Technology Scaling Runs Out of Steam Transistor counts increase exponentially, but… © Hardavellas 3 Can no longer feed all cores with data fast enough (package pins do not scale) Can no longer power the entire chip (voltage, cooling do not scale) Power Wall

4. Technology Scaling Runs Out of Steam Transistor counts increase exponentially, but… © Hardavellas 4 Can no longer feed all cores with data fast enough (package pins do not scale) Bandwidth Wall Can no longer keep costs at bay (process variation, defects) Low Yield Can no longer power the entire chip (voltage, cooling do not scale) Power Wall Monolithic (single-chip) processor designs running out of steam too

5. Galaxy: Optically-Connected Disintegrated Processors Physical constraints limit the performance of single-chip designs  Area, Yield, Power, Bandwidth Multi-chip designs break free of these limitations  Processor disintegration  Macro-chip integration © Hardavellas 5 [WINDS-2010, ICS-2014]

6. Outline Introduction ➔ Background Scalable Multi-Chip System Design with Silicon Photonics  Galaxy Architecture  Experimental Results Energy-Proportional Photonic Interconnects  EcoLaser  ProLaser Conclude © Hardavellas 6

7. Nanophotonic Components © Hardavellas 7 off-chip laser source coupler resonant modulators resonant detectors Ge-doped waveguide

8. Modulation and Detection © Hardavellas 8 11010101 10001011 16 - 64 wavelengths DWDM 10Gbps per link 5 - 20μm waveguide pitch 1 - 16 TB/s/mm bandwidth density

9. Outline Introduction Background Scalable Multi-Chip System Design with Silicon Photonics ➔ Galaxy Architecture  Experimental Results Energy-Proportional Photonic Interconnects  EcoLaser  ProLaser Conclude © Hardavellas 9

10. Optical Crossbar © Hardavellas 10 Cluster 0 Cluster 1 Cluster 2 Cluster 3 Cluster 0 Cluster 1 Cluster 2 Cluster 3

11. Routing Example © Hardavellas 11 Optical Fiber bundle Waveguide bundle AB

12. Galaxy Architecture (5-chiplet example) © Hardavellas 12

13. Why Fibers? Traditional alternatives are: Electrical strips (SerDes) on FR4 board  Fibers are 10x more efficient: 180 fJ/bit vs. 2.5pJ/bit for 4’’  Fibers offer 8 TB/s/mm vs. pin interface (<200GB/s) Electrical wires on a silicon interposer  Fibers are 3x more efficient: 180 fJ/bit vs. 0.5pJ/bit  Fibers have a reach of several feet, vs. ~4 mm  Fibers transmit one bit per 4-16 um pitch, vs ~70 um pitch SOI waveguides on a silicon wafer  Fibers are twice as fast: 0.286c vs 0.676c  Fibers have negligible optical loss: 0.3db/cm vs. 0.2db/Km Do not confine the design on a single board, package, or wafer © Hardavellas 13

14. Dense Off-Chip Coupling Dense optical fiber array [Lee, OSA/OFC/NFOEC 2010] ~3.8dB loss, 8 Tbps/mm demonstrated Misalignment within  loss <1 dB © Hardavellas 14 Connects a fiber array to an on-chip waveguide array at a chip’s edge

15. Outline Introduction Background Scalable Multi-Chip System Design with Silicon Photonics  Galaxy Architecture ➔ Experimental Results Energy-Proportional Photonic Interconnects  EcoLaser  ProLaser Conclude © Hardavellas 15

16. Modeling Infrastructure © Hardavellas 16 3D-stack model SimFlex sampling 95% confidence photonic-layer ring heating

17. Impact of Disintegration: Speedup Over Single-Chip © Hardavellas 17 Processor Disintegration with Galaxy: 2–3x speedup M=Concentrated Mesh w/Exp.Links, C=Corona, F=Firefly, G=Galaxy

18. Impact of Disintegration: Speedup Over “Unlimited” © Hardavellas 18 Galaxy matches the performance of “unlimited” designs M=Concentrated Mesh w/Exp.Links, C=Corona, F=Firefly, G=Galaxy

19. Macrochip Integration with Galaxy © Hardavellas 19 Fiber Galaxy: 2.5x speedup over Oracle Macrochip (6.8x max) Galaxy’s lasers each consumes 6x less power

20. 80-core 5-chiplet Galaxy Thermal CFD Modeling © Hardavellas 20 8cm spacing allows cooling with cheap passive heatsinks 88.2 0 C

21. 9-chiplet Dense Array (Oracle Macrochip) © Hardavellas 21 Tight arrangement points to liquid cooling requirement 249 0 C

22. 9-chiplet Galaxy 3D © Hardavellas 22 Flexible fibers allow “virtual chip” to break free of 2D planar designs 83.6 0 C

23. Galaxy Summary “Virtual chips” with the performance of unlimited designs Breaks free of typical physical constraints  Large aggregate area  Improved yield (break-even point : 60% yield for photonics)  Tb/s/mm bandwidth density  Pushes back power wall Processor disintegration  2.4x – 3.2x avg. speedup (3.4 max)  2.4x – 2.8x avg. smaller EDP (7.1x max) Macrochip integration  2.5x speedup over Oracle Macrochip (6.8x max)  6x more power efficient links © Hardavellas 23

24. © Hardavellas 24 Problem 1: High Laser Power Silicon photonics are emerging as a promising technology for high-bandwidth, low-latency, and energy-efficient communication in many-cores However, lasers are really power-hungry  Optical devices induce optical loss (13+ dB is typical)  WDM-compatible lasers are 5-10% efficient 10-20x higher laser power than required optical output

25. Problem 2: Laser Power is Wasted Interconnect may stay idle for long times  Compute-intensive execution phases of workloads  30% server utilization in Google data centers But laser stays always on!  …even during periods of interconnect inactivity © Hardavellas 25 Up to 88% energy waste in real-world workloads

26. Solution: Laser Power Gating Turn the lasers off when interconnect is idle Turn the lasers on before sender transmits Overlooked until now  Traditional comb lasers are slow to turn on New enabling technology: Germanium Lasers  Turn on/off in 1ns  On-chip  simplify design and lower cost © Hardavellas 26

27. Outline Introduction Background Scalable Multi-Chip System Design with Silicon Photonics  Galaxy Architecture  Experimental Results Energy-Proportional Photonic Interconnects ➔ EcoLaser  ProLaser Conclude © Hardavellas 27

28. EcoLaser: Adapt Laser to Interconnect Traffic First paper on laser power gating  Power down lasers when not needed  Relaxed turn-off to facilitate opportunistic senders Adaptive mechanism to determine stay-on time  Monitors interconnect activity Result Highlights  24 – 77% energy savings on real workloads  1.1 – 2x speedup  Within 2-6% of a perfect (ideal) scheme © Hardavellas 28 [ISLPED-2014]

29. SWMR Optical Bus © Hardavellas 29 1101010110001011 1101010110001011 11 11 11 11 … … Router 0 (Home) Router 1Router N-2Router N-1 … Data Bus Reservation Channel R0R0 R1R1 D0D0 D1D1 ……

30. MWSR Optical Bus © Hardavellas 30 1101010110001011 1101010110001011 1 … … Router 0Router 1 Router N-2 Router N-1 (Home) … Token Stream Data Bus T0T0 T1T1 D0D0 D1D1 ……

31. EcoLaser Design - MWSR Laser turn-on request via token stream  Laser Turn On 31 © Hardavellas

32. 32 Adaptive Laser Control The laser stays on for K cycles each time it turns on Static-K laser control  K is statically set, stays fixed across time  We model a range of static schemes Adaptive laser control  Approximate ideal value of K at each time interval  Monitor the laser turn-on signals  Too many  increase K  higher performance  Too few  lower K  higher energy savings Balance energy savings with interconnect performance

33. Interconnect Performance - MWSR © Hardavellas 33 Static: saturate early (56% throughput for Static-1) Adaptive: provides max interconnect throughput

34. © Hardavellas 34 Static: fail to capture all energy savings Adaptive: within 3% of the Perfect scheme Interconnect Energy - MWSR

35. © Hardavellas 35 Higher laser power -> higher performance impact Adaptive: 2x speedup at 29% laser energy (within 6% Perfect) EcoLaser Speedup – radix-64 MWSR Measured Injection Rate

36. © Hardavellas 36 Impact on Energy × Delay – radix-64 MWSR... Radix-64 impractical to implement without laser control Adaptive: 3.8x lower EDP, within 7% of Perfect

37. EcoLaser Summary Power down lasers when not needed  Relaxed turn-off to facilitate opportunistic senders  Monitor & adapt to interconnect activity Result Highlights  24 – 77% energy savings on real workloads  1.1 – 2x speedup  Within 2-6% of a perfect (ideal) scheme But  Complicated token scheme  Can do much better © Hardavellas 37 Yes, we can improve 2x over this “Perfect” scheme

38. Outline Introduction Background Scalable Multi-Chip System Design with Silicon Photonics  Galaxy Architecture  Experimental Results Energy-Proportional Photonic Interconnects  EcoLaser ➔ ProLaser Conclude © Hardavellas 38

39. DFB Laser … λ1λ1 λ2λ2 λNλN Data-Only Bits λ 1 λ 2 … λ N … DFB Laser … λ1λ1 λ2λ2 λNλN Common Bits λ 1 λ 2 … λ N … Data Bus ProLaser: Segregate Data from Control 39 © Hardavellas Switch on only the necessary interconnect portion [IEEE Photonics - 2014]

40. ProLaser: Proactively Switch On Laser © Hardavellas 40 L2 Cache Requests & Replies Switch Allocator & VC Allocator … … Reservation Channels Data Channels … L L L L L L R1R1 R2R2 RNRN CH 1 CH N CH 2 RCH N RCH 2 RCH 1 Lasers Inject 1 Inject C … … Eject 1 Eject C Data Channel i Data Channel 1 Data Channel N O/E Laser Controller VC0 VC1 VC2 VC0 VC1 VC2 E/O Reservation Channel i … E/O Common Channel i L2 Cache Slice Bloom Filter Bloom filters + coherence protocol  predict accesses

41. ProLaser: Interconnect Performance © Hardavellas 41 ProLaser almost perfect saturation; EcoLaser saturates early

42. ProLaser: Interconnect Energy © Hardavellas 42 ProLaser saves 49-88% of laser power ProLaser is ~2x better than EcoLaser; 2-6% of Perfect

43. ProLaser: Performance Impact © Hardavellas 43 60% speedup over No-Ctrl; 40% over flattened buttefly

44. Sensitivity to Laser Turn-on Delay © Hardavellas 44 Tolerates high laser delays (7x increase  15% penalty)

45. Conclusions Galaxy breaks free of typical physical constraints  “Virtual chips” with the performance of unlimited designs  Processor disintegration: 3.2x speedup, 2.8x EDP (7x max)  Macrochip integration: 6.8x speedup, 6x lower power  Provides system design flexibility Adaptive Laser Control  Makes power-hungry photonic interconnects practical  Saves 49-88% of the laser energy  Provides 50-70% speedup © Hardavellas 45

46. Thank You! Questions? © Hardavellas 46

47. TECHNOLOGY BACKUP SLIDES © Hardavellas 47

48. Chip Power Scaling © Hardavellas 48 Chip power does not scale [Azizi 2010]

49. Demand for High-Performance Computing Grows Large Hadron Collider in March’11: 1.6PB data (Tier-1) Large Synoptic Array Survey Telescope: 30 TB/night  i.e., 2x Sloan Digital Sky Surveys/night  Sloan: more data than entire history of astronomy before it Data grows faster than Moore’s Law © Hardavellas 49 More data  more computing power to process them

50. Voltage Scaling Has Slowed © Hardavellas 50 In last decade: 13x transistors but 30% lower voltage Cannot run all transistors fast enough

51. Pin Bandwidth Scaling © Hardavellas 51 [TU Berlin] Cannot feed cores with data fast enough to keep them busy

52. Electrical vs. Photonic Links © Hardavellas 52 [Nitta et al., 2013]

53. Electrical (SerDes) vs. SOI Waveguides vs. Fibers © Hardavellas

54. SWMR vs. MWSR Crossbar © Hardavellas 54 Single-Writer Multiple-Reader Broadcast bus All receivers always read On-rings  optical loss High laser power Multiple-Writer Single-Reader Only one receiver reads Only one ring is on  low loss Low laser power Needs arbitration

55. GALAXY BACKUP SLIDES © Hardavellas 55

56. Single Chiplet Connectivity © Hardavellas 56

57. Galaxy MWSR Optical Crossbar © Hardavellas 57 MWSR avoids broadcast data bus, but requires arbitration

58. Token-Based Arbitration © Hardavellas 58 8 cycles on average for token arbitration (5 chiplets)

59. Modeling Infrastructure © Hardavellas 59 3D-stack model SimFlex sampling 95% confidence photonic-layer ring heating

60. Architectural Parameters © Hardavellas 60

61. Nanophotonic Parameters © Hardavellas 61

62. Load Latency (uniform random traffic) © Hardavellas 62

63. Load-Latency Curves © Hardavellas 63 16 tokens provide optimal buffer depth

64. Impact of Disintegration: Speedup Over “Unlimited” © Hardavellas 64 Galaxy matches the performance of “unlimited” designs M=Concentrated Mesh w/Exp.Links, C=Corona, F=Firefly, G=Galaxy

65. Performance Against “Realistic” Designs © Hardavellas 65 Realistic: within power and bandwidth envelopes Galaxy chiplets within 66 - 88 o C  chiplets run at max speed Galaxy: 2.4x - 3.2x speedup on average (3.4 max) Galaxy: 2.4x-2.8x smaller EDP on average (up to 7.1x smaller)

66. Comparison Against Multi-Chip Alternatives © Hardavellas 66

67. Tapered vs. Optical Proximity Couplers © Hardavellas 67 6x less laser power than Oracle Macrochip with demonstrated couplers

68. Laser Power Sensitivity to Optical Parameters © Hardavellas 68 Coupler Loss Off-Ring Loss Waveguide & Filter Drop Loss Modulator Insertion Loss Highly sensitive to coupler loss, insensitive to other losses

69. Sensitivity to Fiber Density 116mm 2 chiplets  43mm along the chip edge Enough room for 172 fibers @ 250μm pitch © Hardavellas 69 128 fibers: within 3% of max performance

70. Energy-Delay Product © Hardavellas 70 Galaxy: 2.4x-2.8x smaller EDP on average (7.1x max)

71. Energy per Instruction © Hardavellas 71 Galaxy: 12-20% lower energy/instruction on average (up to 2.3x less)

72. 9-chiplet Galaxy 2D © Hardavellas 72 Cooling 9 chiplets with passive heatsinks 110 0 C

73. ECOLASER BACKUP SLIDES © Hardavellas 73

74. Laser Power Consumption Modulator Insertion Loss Off-Ring Loss Waveguide Loss Filter Drop Loss 10x Wall- plug Laser Power 74 © Hardavellas

75. EcoLaser Design - SWMR Message in injection buffers  Laser Turn On 75 © Hardavellas

76. EcoLaser Token Design Traditional token provides arbitration only  1 bit is sufficient EcoLaser token needs to  T: Facilitate arbitration  L: Indicate light presence on data bus  S: Provide laser turn-on signal  Check if the laser is on first, before sending the turn on signal  Laser turn-on signal should trail T/L by one cycle  Denote dedicated slot (to avoid starvation) © Hardavellas 76 T L S T

77. EcoLaser 3-bit Token and Laser Controller FSM © Hardavellas 77

78. EcoLaser Writer Node FSM © Hardavellas 78

79. MWSR Laser Control Example Token stream Data stream R3R3 R2R2 R1R1 R0R0 Router Laser Source R0R0 R1R1 R2R2 R3R3 T3T3 11 0 T2T2 11 0 T1T1 11 0 T0T0 11 0 T7T7 11 0 T6T6 11 0 T5T5 11 0 T4T4 11 0 D6D6 D5D5 D4D4 D3D3 D2D2 D1D1 D0D0 D7D7 Off 79 © Hardavellas

80. R0R0 R1R1 R2R2 R3R3 T4T4 11 0 T3T3 11 0 T2T2 11 0 T1T1 11 0 T0T0 11 0 T7T7 11 0 T6T6 11 0 T5T5 11 0 D7D7 D6D6 D5D5 D4D4 D3D3 D2D2 D1D1 D0D0 Off t = 1 MWSR Laser Control Example 80 © Hardavellas

81. 0 R0R0 R1R1 R2R2 R3R3 T5T5 11 0 T4T4 11 0 T3T3 11 0 T2T2 11 0 T1T1 11 0 T0T0 11 0 T7T7 0 0 T6T6 11 0 D0D0 D7D7 D6D6 D5D5 D4D4 D3D3 D2D2 D1D1 Off t = 2 MWSR Laser Control Example 81 © Hardavellas

82. 0 R0R0 R1R1 R2R2 R3R3 T6T6 11 0 T5T5 11 0 T4T4 11 0 T3T3 11 0 T2T2 11 0 T1T1 11 0 T0T0 0 0 T7T7 10 0 D1D1 D0D0 D7D7 D6D6 D5D5 D4D4 D3D3 D2D2 Off t = 3 MWSR Laser Control Example 82 © Hardavellas

83. 01 R0R0 R1R1 R2R2 R3R3 T7T7 11 0 T6T6 11 0 T5T5 11 0 T4T4 11 0 T3T3 11 0 T2T2 11 0 T1T1 0 T0T0 10 0 D2D2 D1D1 D0D0 D7D7 D6D6 D5D5 D4D4 D3D3 On t = 4 MWSR Laser Control Example 83 © Hardavellas

84. R0R0 R1R1 R2R2 R3R3 T0T0 11 0 T7T7 11 0 T6T6 11 0 T5T5 11 0 T4T4 11 0 T3T3 11 0 T2T2 0 T1T1 0 D3D3 D2D2 D1D1 D0D0 D7D7 D6D6 D5D5 D4D4 On t = 5 MWSR Laser Control Example 0110 84 © Hardavellas

85. R0R0 R1R1 R2R2 R3R3 T1T1 11 0 T0T0 11 0 T7T7 11 0 T6T6 11 0 T5T5 11 0 T4T4 11 0 T3T3 0 T2T2 0 D4D4 D3D3 D2D2 D1D1 D0D0 D7D7 D6D6 D5D5 On t = 6 MWSR Laser Control Example 0110 85 © Hardavellas

86. 0110 R0R0 R1R1 R2R2 R3R3 T2T2 11 0 T1T1 11 0 T0T0 11 0 T7T7 11 0 T6T6 11 0 T5T5 11 0 T4T4 0 T3T3 0 D5D5 D4D4 D3D3 D2D2 D1D1 D0D0 D7D7 D6D6 On t = 7 MWSR Laser Control Example 86 © Hardavellas

87. R0R0 R1R1 R2R2 R3R3 T3T3 10 1 T2T2 11 0 T1T1 11 0 T0T0 11 0 T7T7 11 0 T6T6 11 0 T5T5 T4T4 D6D6 D5D5 D4D4 D3D3 D2D2 D1D1 D0D0 D7D7 On t = 8 MWSR Laser Control Example 0110 00 87 © Hardavellas

88. R0R0 R1R1 R2R2 R3R3 T4T4 11 1 T3T3 10 1 T2T2 11 0 T1T1 11 0 T0T0 11 0 T7T7 11 0 T6T6 T5T5 D7D7 D6D6 D5D5 D4D4 D3D3 D2D2 D1D1 D0D0 On t = 9 MWSR Laser Control Example 0110 00 88 © Hardavellas

89. R0R0 R1R1 R2R2 R3R3 T5T5 11 1 T4T4 11 1 T3T3 10 1 T2T2 11 0 T1T1 11 0 T0T0 11 0 T7T7 T6T6 D0D0 D7D7 D6D6 D5D5 D4D4 D3D3 D2D2 D1D1 On t = 10 MWSR Laser Control Example 0110 00 89 © Hardavellas

90. R0R0 R1R1 R2R2 R3R3 T6T6 11 1 T5T5 11 1 T4T4 11 1 T3T3 10 1 T2T2 11 0 T1T1 11 0 T0T0 T7T7 D1D1 D0D0 D7D7 D6D6 D5D5 D4D4 D3D3 D2D2 OnOn t = 11 MWSR Laser Control Example 0110 00 90 © Hardavellas

91. R0R0 R1R1 R2R2 R3R3 T7T7 11 1 T6T6 11 1 T5T5 11 1 T4T4 11 1 T3T3 10 1 T2T2 11 0 T1T1 T0T0 D2D2 D1D1 D0D0 D7D7 D6D6 D5D5 D4D4 D3D3 On t = 12 MWSR Laser Control Example 0110 00 91 © Hardavellas

92. R0R0 R1R1 R2R2 R3R3 T0T0 11 1 T7T7 11 1 T6T6 11 1 T5T5 11 1 T4T4 11 1 T3T3 10 1 T2T2 T1T1 D3D3 D2D2 D1D1 D0D0 D7D7 D6D6 D5D5 D4D4 On t = 13 MWSR Laser Control Example 0110 00 92 © Hardavellas

93. R0R0 R1R1 R2R2 R3R3 T1T1 11 0 T0T0 11 1 T7T7 11 1 T6T6 11 1 T5T5 10 0 T4T4 11 1 T3T3 T2T2 D4D4 D3D3 D2D2 D1D1 D0D0 D7D7 D6D6 D5D5 On t = 14 MWSR Laser Control Example 0110 00 93 © Hardavellas

94. 01 R0R0 R1R1 R2R2 R3R3 T2T2 11 0 T1T1 11 0 T0T0 11 1 T7T7 11 1 T6T6 11 1 T5T5 10 0 T4T4 1 T3T3 11 0 D5D5 D4D4 D3D3 D2D2 D1D1 D0D0 D7D7 D6D6 Of f t = 15 MWSR Laser Control Example 94 © Hardavellas

95. R0R0 R1R1 R2R2 R3R3 T3T3 11 0 T2T2 11 0 T1T1 11 0 T0T0 11 1 T7T7 11 1 T6T6 11 1 T5T5 10 0 T4T4 11 1 D6D6 D5D5 D4D4 D3D3 D2D2 D1D1 D0D0 D7D7 Off t = 16 MWSR Laser Control Example 95 © Hardavellas

96. R0R0 R1R1 R2R2 R3R3 T4T4 11 0 T3T3 11 0 T2T2 11 0 T1T1 11 0 T0T0 11 1 T7T7 11 1 T6T6 11 1 T5T5 10 0 D7D7 D6D6 D5D5 D4D4 D3D3 D2D2 D1D1 D0D0 Off t = 17 MWSR Laser Control Example 96 © Hardavellas

97. 1 R0R0 R1R1 R2R2 R3R3 T5T5 11 0 T4T4 11 0 T3T3 11 0 T2T2 11 0 T1T1 11 0 T0T0 11 1 T7T7 1 1 T6T6 11 1 D0D0 D7D7 D6D6 D5D5 D4D4 D3D3 D2D2 D1D1 Off t = 18 MWSR Laser Control Example 97 © Hardavellas

98. EcoLaser Nanophotonic Parameters © Hardavellas 98

99. Interconnect Performance – radix-16 MWSR © Hardavellas 99 Static: saturate early (56% throughput for Static-1) Adaptive: provides max interconnect throughput

100. © Hardavellas 100 Static: fail to capture all energy savings Adaptive: within 3% of the Perfect scheme Interconnect Energy – radix-16 MWSR

101. Interconnect Performance – radix-16 SWMR 101 © Hardavellas

102. Interconnect Energy – radix-16 SWMR 102 © Hardavellas

103. 103 Laser power savings leave more power for cores  faster Adaptive: 1.1x speedup at 50% laser energy (within 2% Perfect) EcoLaser Speedup – radix-16 MWSR Measured Injection Rate

104. © Hardavellas 104 Laser power savings leave more power for cores  faster Adaptive: 5% speedup at 50% laser energy (within 2% Perfect) EcoLaser Speedup – radix-16 MWSR Measured Injection Rate

105. EcoLaser Speedup – Radix-16 SWMR 105 © Hardavellas

106. 106 Higher laser power -> higher performance impact Adaptive: 2x speedup at 29% laser energy (within 6% Perfect) EcoLaser Speedup – radix-64 MWSR Measured Injection Rate

107. EcoLaser Speedup for Radix-64 MWSR © Hardavellas 107 EcoLaser Power Savings  ~2x Speedup

108. EcoLaser Speedup – Radix-64 SWMR 108 © Hardavellas

109. EcoLaser Speedup for Radix-64 SWMR © Hardavellas 109 EcoLaser Power Savings  ~2x Speedup

110. Static-1 is 19% slower than No-Ctrl on average (30% maximum). Adaptive saves 45% laser energy and it is 4.8% slower than Perfect. Impact of Latency Overhead 110 © Hardavellas

111. Impact of Latency Overhead 111 © Hardavellas

112. Impact of Latency Overhead 112 © Hardavellas

113. Impact of Latency Overhead 113 © Hardavellas

114. Energy × Delay – radix-16 MWSR No-Ctrl: more energy efficient than Static-1, Power_Eq Adaptive: 13% lower EDP, within 2% of Perfect... 114 © Hardavellas

115. Energy × Delay – radix-16 SWMR 115 © Hardavellas

116. 116 Impact on Energy × Delay – radix-64 MWSR... Radix-64 impractical to implement without laser control Adaptive: 3.8x lower EDP, within 7% of Perfect

117. Energy × Delay – radix-64 SWMR 117 © Hardavellas

118. Backup Slides Why not use Off-Chip Laser?  Pro: Higher eff. & off the chip power budget  Con: Coupler Loss and intrinsic loss* Conclusion: Off-chip laser source might increase the total system power consumption. On-Chip laser source with control is more efficient than off- chip lasers. Ge-based lasers manufactured footprint 1.6um x 4mm, could be smaller. 118 © Hardavellas

119. Experimental Methodology CMP Size64 cores, 480 mm2 Processing CoreULTRASPARC III ISA, up to 5Ghz, OoO, 4-wide dispatch/retirement, 96-entry ROB L1 CacheSplit I/D, 64KB 2-way, 2-cycle load-to-use, 2 ports, 64-byte blocks, 32 MSHRs, 16-entry victim cache L2 CacheShared, 512 KB per core, 16 way, 64-byte blocks, 14 cycle-hit, 32 MSHRs, 16-entry victim cache Memory ControllerOne per 4 cores, 1 channel per Memory Controller Round-robin page interleaving Main MemoryOptically connected memory [2], 10ns access NetworkSWMR and MWSR crossbars, radix-16 and -64 300-bit wide links @ 10GHz, 20 flit deep buffers, 3 cycle router delay 119 © Hardavellas

120. Radix-16Radix-64 DWDM6416 WG Loss3 dB Non-Linearity1 dB Modulator Ins.0.5 dB Ring Through10.24 dB Filter Drop1.2 dB Photodetector0.1 dB Total Loss16.04 dB Laser Power0.401 mW Total Laser Power 20.1W78.1W Laser Power Consumption 120 © Hardavellas

121. Radix-16AreaRadix-64 DWDM6416 WG80160 mm21200300 mm2 Ring Resonators 77K7.7 mm21.2 M100 mm2 Lasers480034 mm219K125 mm2 Optical Component Count 121 © Hardavellas

122. Workloads Fmm: Input 128K Moldyn: 15, 20, 3.2 M Barnes: Input 64K Tomcatv: 4096, 10 Appbt: in.24x24x24x8bit Ocean: 1026, 9600 Em3d: 400K, 2, 15, 5 122 © Hardavellas

123. PROLASER BACKAUP SLIDES © Hardavellas 123

124. Data-Only Bits DFB … … Laser Switch λ1λ1 λ2λ2 λNλN λ1λ1 λ2λ2 λNλN λ 1 … λ N λ 1 λ 2 … λ N λ & λ … … Common Bits Data Bus Network-on-chip Off-chip laser die Optical Fiber SOI Waveguides Off-Chip Ge-based Laser Source

125. ProLaser – Architectural Parameters © Hardavellas 125

126. ProLaser – Nanophotonic Parameters © Hardavellas 126