1. 1 DB 1.3 Case Study: eBay’s 40G/100G Network Upgrade Shawn Tugwell, Sr. Data Center Design Engineer – eBay Gary Bernstein, RCDD, CDCD Sr. Director of Product Management - Leviton

2. 2 Data Center World – Certified Vendor Neutral Each presenter is required to certify that their presentation will be vendor-neutral. As an attendee you have a right to enforce this policy of having no sales pitch within a session by alerting the speaker if you feel the session is not being presented in a vendor neutral fashion. If the issue continues to be a problem, please alert Data Center World staff after the session is complete.

3. 3 3 Key Things you will Learn During this Session 1.Get an understanding of what a real-world 40G line-rate design at a web-scale data center looks like 2.The importance of collaboration between a customer and vendor 3.The importance of designing a flexible network that can adapt to different technologies

4. 4 eBay Snapshot Revenue of $22 Billion 162 million users 800 million items listed on eBay marketplace 250 million+ searches every day 20,000 packages processed daily by eBay shipping center Source: Digital Marketing Ramblings (DMR), Dec. 2015

5. 5 eBay Data Centers... US Locations: South Jordan, UT Phoenix, AZ Chandler, AZ Las Vegas, NV Reno, NV European locations: Luxembourg, Ireland, Australia

6. 6 Those Network Guys…. They told me they wanted: 1.A network to support next generation servers at line-rate speeds 2.A Leaf-Spine Network Architecture using high density switches 3.A converged network to support LAN & SAN 4.Use Arista MXP ports to increase density and lower cost

7. 7 What did this mean for the physical layer design? Every rack contains 96 Servers, 2x10G NICs each to TOR 6X increase in port density per server cabinet…means more patching needed Will it work up to 500 meter cabling runs Allow easy migration to 100G & 400G Desire to have 100% fiber utilization

8. 8

9. 9 First Question… Use Multimode or Single-mode Optics? Two primary objectives were 500m reach and cost/channel With the size of eBay’s DCs, use of containers and the need to tie data halls together...eBay typically has runs up to 500m IEEE and manufacturers state a 40G maximum reach of 150m with OM4 MM eBay preferred to use MM (40G-XSR4) vs. SM (40G-PLRL4) transceivers due to 25% lower cost One cabling system stated 300m with OM4+ cabling, so eBay asked to do testing to see if 750m will actually work

10. 10 40G Testing of OM4+ Channels 6 typical eBay channels were set-up in a data center Test plan: Run 40GbE traffic over network and confirm ZERO dropped packets and ZERO errors OM4+ Fiber Cabling 24F MTP trunks included Conversion cassettes 8F MTP trunks MTP array cords 6 MTP connections in the channel

11. 11 40G Testing of OM4+ Channels 24F-3x8F Conversion Cassette 24F MTP in MXP switch port 8F MTP in TOR 40G-XSR4 switch port

12. 12 40G Testing Results Successful ERROR-FREE transmission of 40GbE over 750m of OM4+ Cabling Result: eBay could use the lower cost multimode transceivers

13. 13 The Structured Cabling Solution Needed Very High density - 288 Fiber trunk cables using OM4+ glass 1U Angled patch panel with rear cable management Max density of 1U panel 72 x 10G-SR ports (144 fibers using LC) 72 x 40G-SR4 ports (576 fibers using MTPs) Easy way to distinguish Tx and Rx paths Easy way to know polarity of array cords Did not exist at the time

14. 14 New Product Development Collaboration Both companies worked together to develop a new panel solution that met design requirements

15. 15 Full 40G-Line Rate Channel

16. 16 Network Row- Core Switch Switch with 8 line cards Each with 36 x 40G-XSR4 QSFP+ ports 8F MTP-MTP Array Cords from Core Switch into MTP-MTP adapter in high density panel 288F trunks With 8F MTPs coming from panel

17. 17

18. 18 Network Row- Distribution Switch Switch with 8 line cards Each with 12 x 100G ports MXP port requires 24F MTP connector 24F to 3x8F MTP-MTP Harness from MXP ports into MTP-MTP pass-thru and into 288F trunks with 8F MTPs

19. 19 Patching from Arista MXP Line Card 576 fibers into 1U Patch Panel High-Density Angled 1RU Patch Panel Even number line cards will terminate to the left side of the patch panel as you look at it from the front of the switch, odd on the right

20. 20 Patching from Arista MXP Line Card 576 fibers into 1U Patch Panel

21. 21 Server Row with TOR Switches TOR switch With 32 x QSFP+ 40G-XSR4 ports 8F trunks from Network Row go into MTP pass-thru in high density panel 8F MTP-MTP array cords connect into TOR

22. 22 40G Connections into TOR in Server Cabinet

23. 23 Converting 40G to 10G channels

24. 24 Violet OM4+ Cable used to differentiate from installed OM3

25. 25 Collaboration Thorough understanding of applications Bouncing ideas off each other Back & forth discussion Best in class solutions

26. 26 Cabling Solution used in eBay design 288F OM4+ 8F MTP Fiber Violet Trunks New ultra high density panel Unique polarity array harness Color-coded array cords

27. 27 10/40/100G Migration Solutions 24-fiber MTP backbone cabling provides the most flexibility and ease of migrating from 10G to 40G or 100G networks 8-fiber MTP backbone cabling can be a good option for the right greenfield 40G applications 24-fiber MTP 8-fiber MTP

28. 28 In Summary We delivered a reliable product to our customer – the eBay network team Our operations team was extremely happy we followed the KISS principle Cabling design is very flexible…allowing us to adapt for different applications

29. 29 Thank you