Need for Speed: Beyond 100GbE Moderator: Scott Kipp, President of Ethernet Alliance, Principle Engineer, Brocade Panelist #1: Alan Weckel, Vice President, Dell’Oro group Panelist #2: Dr. Jeffery J. Maki, Distinguished Engineer, Juniper Panelist #3: Dr. Gordon Brebner, Distinguished Engineer, Xilinx

Agenda Introductions: Scott Kipp, Moderator Panelist #1: Alan Weckel, 10, 40 and 100GbE Deployments in the Data Center Panelist #2: Dr. Jeffery J. Maki, Stepping Stones to Terabit-Class Ethernet Panelist #3: Dr. Gordon Brebner, Technology Advances in 400GbE Components Q&A 2:40 – Live Broadcast from IEEE 802.3 Meeting in Orlando from John D’Ambrosia Update on 400GbE Call For Interest © 2012 Ethernet Alliance

Disclaimer The views WE ARE expressing in this presentation are our own personal views and should not be considered the views or positions of the Ethernet Alliance.

More Rich Media Content Bandwidth Growth Source: nowell_01_0911.pdf citing Cisco Visual Networking Index (VNI) Global IP Traffic Forecast, 2010–2015, http://www.ieee802.org/3/ad_hoc/bwa/public/sep11/nowell_01_0911.pdf More Devices More Internet Users More Rich Media Content Key Growth Factors Broadband 2010- 7Mbps 2015 – 28 Mbps 15B Devices In 2015 Increased # of Users Increased Access Rates and Methods Increased Services + = Bandwidth Explosion Everywhere Speed Increasing 3B Users In 2015 2010- 1 Minute video 2015 – 2 hour HDTV Movie

Bandwidth Growth Vs Ethernet Speeds IP Traffic is growing ~ 30%/year If 400GbE is released in 2016, Ethernet speeds will grow at about 26%/year Internet traffic would grow ~10X by 2019 at 30%/year Ethernet speeds to grow 4X by 2016 at 26%/year Internet traffic normalized Ethernet Speed (Gb/s) to 100 in 2010

Ethernet Optical Modules CFP CFP2 300 Pin MSA XENPAK XPAK X2 100G 10G 1G 100GbE CFP4 XFP CXP 40GbE QSFP28 40G Key: Ethernet Standard Released Module Form Factor Released QSFP+ Data Rate and Line Rate (b/s) 10GbE SFP+ GBIC SFP GbE 1995 2000 2005 2010 2015 Standard Completed

Ethernet Speeds 2010-2025 1T Data Rate and Line Rate (b/s) 400G 100G If Ethernet line rates doubles the line rate every 3 years at 26% CAGR, then 400GbE would come out in 2016 and TbE would come out in 2020. Something will have to change. Key: Ethernet Speeds Ethernet Electrical Interfaces Hollow Symbols = predictions Stretched Symbols = Time Tolerance 400GbE 4X100G 100GbE 1X100G TbE 10X100G nX100G 1.6TbE 16X100G 1T 100G 10G 16x25G 400GbE 16X25G 8X50G 400GbE 400G 100GbE 10X10G 4x25G 100GbE 4X25G Data Rate and Line Rate (b/s) 40GbE 4X10G 40G 4x10G 10X10G 2010 2015 2020 2025 Standard Completed

Ethernet Success Ethernet has been extremely successful at lowering the price/bit of bandwidth If the cost of a new speed/technology is too high, then it is not widely deployed Technology needs to be ripe for picking 400GbE is ripe with 100GbE technology TbE isn’t ripe and a revolutionary breakthrough would be needed to get it before 2020 This panel will look at how high speeds of Ethernet are being deployed and the technology that is leading to the next generation of Ethernet

10, 40 and 100GbE Deployments in the Data Center Alan Weckel Vice President, Data Center Research Dell’Oro Group

Introduction Progress on server migration from 1 GbE to 10 GbE 10G Base-T update Data center networking market update 40 GbE and 100 GbE market forecasts

Overview Dell’Oro Group is a market research firm that has been tracking the Ethernet Switch and Routing markets on a quarterly basis since 1996 We also track the SAN market, Optical market, and most Telecom equipment markets We produce quarterly market share reports that include port shipments as well as market forecasts

Data Center Bandwidth Shipping – Ethernet Switching Petabytes per Second Shipped per Year

Switch Attach Rate on Servers 1 GbE 10 GbE 40 GbE Percent of Server Shipments

Data Center Port Shipments – 10 G Base-T Port Shipments 10G Base-T controller and adapter ports Port Shipments in Thousands 10G Base-T switch ports

Data Center Port Shipments – Ethernet Switching Port Shipments in Millions

Data Center Port Shipments – Ethernet Switching Port Shipments in Millions

Summary Ethernet Switches will be responsible for the majority of 40 GbE and 100 GbE port shipments over the next five years Form-factor and cost driving 40 GbE over 100 GbE 10 GbE server access transition is key to higher speed adoption

Jeffery J. Maki Distinguished Engineer, Optical Juniper Networks, Inc. Stepping Stones to Terabit-Class Ethernet: Electrical Interface Rates and Optics Technology Reuse Jeffery J. Maki Distinguished Engineer, Optical Juniper Networks, Inc.

100G

CFP, CFP2 and CFP4 for SMF or MMF Applications CFP MSA Form Factors: http://www.cfp-msa.org/ CFP CFP2 CFP4 CFP4(LC) CFP2(LC) CFP(LC) Optical Connector LC Duplex (depicted) MPO Courtesy of TE Connectivity

Module Electrical Lane Capability CFP CFP2 CFP4 12x10G electrical lanes 10x10G or 8x25G electrical lanes 4x25G electrical lanes CAUI for 10x10G CPPI & CAUI for 10x10G CAUI-4 for 4x25G CAUI-4 for 4x25G

CFP, CFP2, and CFP4 for 100G Ethernet SMF PMD Transmit side only depicted. Current Options Up to 10 km: 100GBASE-LR4 Up to 40 km: 100GBASE-ER4 CFP LAN WDM 1295.56 nm Gear Box 1300.05 nm 1304.58 nm 1309.14 nm LAN WDM CFP2 1295.56 nm Gear Box 1300.05 nm 1304.58 nm 1309.14 nm CFP4 4 λ on LAN WDM

400G

Projection of Form Factor Evolution to 400G CD-CFP4 defensible speculation CD-CFP2 CFP CD-CFP CFP2 CFP4 16x25G electrical lanes 8x50G electrical lanes 4x100G electrical lanes Roman Numerals XL = 40 C = 100 CD = 400 CFP4

Likely MSA Activity CFP MSA http://www.cfp-msa.org/ CD-CFP: Current CFP needs revamping to support 16 x 25G CD-CFP2: Current CFP2 is ready for 8 x 50G CD-CFP4: Unclear New CDFP MSA http://www.cdfp-msa.org/ High-density form factor supporting 16 x 25G From slide 26 of http://www.ieee802.org/3/cfi/0313_1/CFI_01_0313.pdf

400G Optics Requirements First-generation transceivers have to be implementable that meet and eventually do better than these requirements Size (Width):  82 mm (CFP width, ~4 x CFP4) Cost:  4 x CFP4 Power:  24 W (4 x 6 W power profile of CFP4) Improved bandwidth density transceivers will need higher rate electrical-lane technology 50G 100G

How 400G Ethernet Can Leverage 100G Ethernet 100G Ethernet up to 10 km Duplex Single-Mode Fiber Infrastructure CFP4-LR4 CFP4-LR4 400G Ethernet up to 10 km Parallel Single-Mode Fiber Infrastructure Only 8 Fibers Used CFP4-LR4 CFP4-LR4 CFP4-LR4 CFP4-LR4 CFP4-LR4 CFP4-LR4 CFP4-LR4 CFP4-LR4

Possible SMF Ethernet Road Map: 100G, 400G, 1.6T Early Adopter 400G Mature 400G Early Adopter 1.6T 4 x 100GBASE-LR4 or “400GBASE-PSM4” 4 x 400GBASE-??? or “1600GBASE-PSM4” 400GBASE-??? CFP4(LC) CD-CFP4(LC) CD-CFP2(LC) CFP4(LC) CD-CFP4(LC) CFP4(LC) CD-CFP4(LC) CD-CFP4(LC) CFP4(LC) CD-CFP4(LC) CD-CFP(MPO) Parallel Single Mode, 4 Lanes (PSM4) 4, Tx Fibers and 4, Rx Fibers 1x12 MPO Connector CD-CFP2(MPO) (High-Density 100GE)

Early Adopter 400G using SMF Structured Cabling Technology Reuse: 4 x 100GBASE-LR4 Parallel SMF: “400GBASE-PSM4” Courtesy of Commscope

Early Adopter 400G using MMF Structured Cabling Courtesy of Commscope Technology Reuse: 4 x 100GBASE-SR4 Parallel MMF: “400GBASE-SR16” Parallel Multi-Mode 100GBASE-SR4, 4 x 25G optical lanes: 4, Tx Fibers and 4, Rx Fibers using 1x12 MPO “400GBASE-SR16”, 16 x 25G optical lanes: 16, TX Fibers and 16, Rx Fibers using 2x16 MPO

MMF Breakout Cables— Enabling 400G Adoption 2 x 16 MPO 2 x 16 MMF MT ferrule 1 x 12 (8 used) MPO 1 x 12 (8 used) MPO 1 x 12 (8 used) MPO Courtesy of USConec 1 x 12 (8 used) MPO

100G Can Build 400G at the Cost of 4 x 100G Technology Reuse: 4 x 100GBASE-LR4 Parallel SMF: “400GBASE-PSM4” Technology Reuse: 4 x 100GBASE-SR4 Parallel MMF: “400GBASE-SR16”

Ethernet PMD Maturity & Possible Obsolescence Early Adopter PMD Parallel Fiber, SMF or MMF Leverage of mature PMD from previous speed of Ethernet Planned obsolescence Implementation (with MPO connector) persists as high-density support of previous speed of Ethernet (e.g., 4 x 100G) Mature PMD SMF: Duplex SMF cabling (e.g., with LC duplex connector) MMF: Lower fiber count MMF cabling

SMF Density Road Map 4 x 16 CD-CFP4(LC) CD-CFP4(LC) (mature) (early adopter) Front-Panel Bandwidth Density (Relative) 8 CD-CFP2(MPO) CD-CFP2(MPO) CD-CFP2(LC) (mature) (early adopter) (mature) 4 CFP4(LC) 4 x CFP4(LC) or CD-CFP(MPO) (early adopter) 2 CFP2(LC) 1 CFP(LC) 100G Port Bandwidth 400G 1.6T

Summary Form-factor road map for bandwidth evolution Early adopter 400G Ethernet by reusing 100G module and parallel cabling, SMF or MMF Need for a new, 2 x 16 MMF MT ferrule Possible common module for 400G Ethernet and high-density (4-port) 100G Ethernet Need for new electrical interface definitions supporting lane rates at 50G 100G

Technology Advances in 400GbE Components Gordon Brebner Distinguished Engineer Xilinx, Inc.

400GbE PCS/MAC Expect first: 16 PCS lanes, each at 25.78125 Gbps Glueless interface to optics Possible re-use of the 802.3ba PCS Other options possible for PCS, maybe native FEC Later: 8 lanes, each at 51.56Gbps Or 4 lanes with 2 bits/symbol at 56Gbaud (e.g. PAM4) Packet size 64 bytes to 9600 bytes Use 100GbE building blocks where possible

Silicon technology Technology nodes (silicon feature size) 130nm, 65nm, 40nm, 28/32nm, 20/22nm, 14/16nm Application-Specific Integrated Circuit (ASIC) Fixed chip Increasingly expensive: need high volumes Best suited to post-standardization Ethernet Field Programmable Gate Array (FPGA) Programmable logic chip Suitable for prototyping and medium volumes Best choice for pre-standardization Ethernet

400GbE line/system bridge Wide parallel data path between blocks CDFP or 4xCFP4 Optical 400GbE PMA/PCS 400GbE MAC Bridge logic 500G Interlaken 40 x 12.5G or 48 x 10G SERDES 16 x 25G SERDES ASIC or FPGA chip Line side System side

400 Gbps and 1 Tbps Ethernet MAC options MAC rate = Width x Clock 400 Gbps and 1 Tbps Ethernet MAC options MAC rate Silicon node Technology Data path width Clock frequency 100 Gbps 45, 40nm ASIC 160 bits 644 MHz FPGA 512 bits 195 MHz 400 Gbps 28, 20nm 400 bits 1 GHz 1024 bits 1536 bits 400 MHz 267 MHz 1 Tbps 20, 14nm 2048 bits 2560 bits 488 MHz

Multiple Packets/Word Bus width Max packets Max EOPs 512 2 1 1024 3 1536 4 512 * n n+1 n Up to 512-bit, only one packet completed Just need to deal with EOP then SOP in word Beyond 512-bit, multiple packets completed Need to add parallel packet processing Must deal with varying EOP and SOP positions

400GbE CRC Example All Ethernet packets carry Cyclic Redundancy Code (CRC) for error detection Computed using CRC-32 polynomial Critical function within Ethernet MAC Requirements Computed at line rate Deal with multiple packets in wide data path Economical with silicon resources

400GbE CRC Prototype Xilinx Labs research project Modular: built out of 512-bit 100G units Computes multiple CRCs per data path word Targeting 28nm FPGA (Xilinx Virtex-7 FPGAs) N-bit data path partitioned into 512-bit sections 512-bit unit CRC results combined to get final CRC results

400GbE CRC Prototype Results: 1024-bit width is feasible for 400GbE Other widths: Less challenging clock frequencies Demonstrate scalability beyond 400GbE Data bus word size 1024-bit 1536-bit 2048-bit Max clock frequency (MHz) 400 381 326 Maximum line rate (Gbps) 409 585 668 Latency (ns) 17.5 18.4 21.5 FPGA resources (slices) 2,888 4,410 5,719

Conclusions Can anticipate 400GbE PCS/MAC standard Ever-increasing rates mean ever-wider internal data path width in electronics Leading to multiple packets per data word Possible to prototype pre-standard PCS/MAC using today’s FPGA technology Demonstrated modular Ethernet CRC block based on 100GbE units Silicon resource scales linearly with line rate