1. Firefly: Illuminating Future Network-on-Chip with Nanophotonics Yan Pan, Prabhat Kumar, John Kim †, Gokhan Memik, Yu Zhang, Alok Choudhary EECS Department Northwestern University Evanston, IL, USA {panyan,prabhat-kumar,g-memik, yu-zhang,a-choudhary} @northwestern.edu † CS Department KAIST Daejeon, Korea jjk12@cs.kaist.ac.kr

2.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan2/25 On-Chip Network Topologies Mesh [MIT RAW] [TILE64] [Teraflops] C-Mesh [Balfour’06] [Cianchetti’09] Crossbar [Vantrease’08] [Kirman’06] Others: Torus[Shacham’07], Flattened Butterfly[Kim’07], Dragonfly[Kim’08], Hierarchical(Bus&Mesh)[Das’08], Clos[Joshi’09], Ring[Larrabee], …… ► Network-on-chip is critical for performance.

3.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan3/25 Signaling technologies ► Electrical signaling –Repeater insertion needed –Bandwidth density (up to 8 Gbps/um) [Chang HPCA‘08] ► Nanophotonics –Bandwidth density ~100 Gbps/ μm !!! [Batten HOTI’08] –Generally distance independent power consumption –Speed of light  low latency Propagation Switching [Cianchetti ISCA’09]

4.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan4/25 Nanophotonic components ► Basic components off-chip laser source coupler resonant modulators resonant detectors Ge-doped waveguide

5.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan5/25 Radius r  Baseline Wavelength Temperature t  Manufacturing error correction Carrier density d  Fast tuning by charge injection Resonant Rings ► Selective –Couple optical energy of a specific wavelength

6.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan6/25 Putting it together ► Modulation & detection – ~100 Gbps/μm bandwidth density [Batten HOTI’08] 11010101 10001011 64 wavelengths DWDM 3 ~ 5μm waveguide pitch 10Gbps per link ~100 Gbps/μm bandwidth density

7.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan7/25 What’s the catch? ► Power Cost –Ring heating –Laser Power –E/O & O/E conversions –Distance insensitive ► For short links (2.5mm) –Nanophotonics –Electrical RC lines with repeater insertion [Batten HOTI’08] [Cheng ISCA’06] ► For long links –Nanophotonics Cost stays the same –Electrical Cost increases

8.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan8/25 Here is the idea …… ► Design an architecture that differentiates traffic. –Use electrical signaling for short links. –Use nanophotonics only for long range traffic. ► What do we gain? –Low latency –High bandwidth density –High power efficiency –Localized arbitration –Scalability

9.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan9/25 Outline ► Motivation ► Architecture of Firefly ► Evaluation ► Conclusion

10.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan10/25 Layout View of 64-core Firefly ► Concentration –4 cores share a router –16 routers

11.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan11/25 Layout View of 64-core Firefly ► Concentration ► Clusters –Electrically connected –Mesh topology –4 routers per cluster –4 clusters Cluster 0 (C0) Cluster 1 (C1) Cluster 3 (C3) Cluster 2 (C2)

12.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan12/25 Layout View of 64-core Firefly ► Concentration ► Clusters ► Assemblies –Routers from different clusters –Optically connected –Logical crossbars

13.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan13/25 Layout View of 64-core Firefly ► Clusters –Electrical CMESH ► Assemblies –Nanophotonic crossbars Nanophotonic Crossbars Efficient nanophotonic crossbars needed!

14.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan14/25 Nanophotonic crossbars † ► Single-Write-Multiple-Read (SWMR) [Kirman’06] (CMXbar † ) –Dedicated sending channel –Multicast in nature –Receiver compare & discard –High fan-out  laser power SWMR Crossbar † † [Joshi NOCS’09]

15.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan15/25 Nanophotonic crossbars † ► Multiple-Write-Single-Read (MWSR) [Vantrease’08] (DMXbar † ) –Dedicated receiving channel –Demux to channel –Global arbitration needed! MWSR Crossbar † † [Joshi NOCS’09]

16.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan16/25 Reservation-assisted SWMR ► Goal –Avoid global arbitration –Reduce power ► Proposed design –Reservation channels Narrow –Multicast to reserve Destination ID Packet length –Uni-cast data packet R-SWMR Crossbar

17.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan17/25 Router Microarchitecture ► Virtual-channel router –Added optical link ports and extra buffer. Dedicated sending channel for all traffic. Separate receiving channels from other clusters.

18.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan18/25 ► Routing –Intra-cluster routing –Traversing optical link Routing (FIREFLY_dest)

19.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan19/25 Firefly – another look ► Clusters –Short electrical links –Concentrated mesh ► Assemblies –Long nanophotonic links –Partitioned crossbars ► Benefits –Traffic locality –Reduced hardware –Localized arbitration –Distributed inter-cluster bandwidth

20.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan20/25 Outline ► Motivation ► Architecture of Firefly ► Evaluation ► Conclusion

21.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan21/25 Evaluation Setup ► Cycle-accurate simulator (Booksim) ► Firefly vs. CMESH, Dragonfly † and OP_XBAR ► Synthetic traffic patterns and traces Electrical Hybrid Optical Hybrid [ † Kim et al, ISCA’08]

22.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan22/25 Load / Latency Curve ► Throughput –Up to 4.8x over OP_XBAR –At least +70% over Dragonfly Bitcomp, 1-cycleUniform, 1-cycle 4.8x70%

23.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan23/25 Energy Breakdown ► Reduced hardware by partitioning –Reduced heating ► Throughput impact ► Locality –34% energy reduction over OP_XBAR with locality

24.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan24/25 Technology Sensitivity ► α is heating ratio and β is laser ratio. ► Firefly favors traffic locality. bitcomptaper_L0.7D7

25.  Motivation  Architecture of Firefly  Evaluation  Conclusion ISCA 2009Yan Pan25/25 Conclusion ► Technology impacts architecture –New opportunities in nanophotonics Low latency, high bandwidth density –Tailored architectures needed ► Firefly benefits from nanophotonics by providing –Power Efficiency Hybrid signaling Partitioned R-SWMR crossbars  Reduced hardware/power –Scalability Scalable inter-cluster bandwidth Low-radix routers/crossbars