The Netflix Open Connect Network DKNOG – 21 March 2013

Today we are in 22 locations across the world Today we are in 22 locations across the world... (16 Primary, 6 “network extensions”)

How do we decide what locations to go to? Where is Netflix presently providing service? Our service is limited by the content licenses that we can buy For example, we are in the Nordic countries, the UK, and Ireland but not France/Germany Can we interconnect with networks that are relevant? Going to a peering point that’s all content providers does no good Can we reduce the cost and improve traffic management for networks who need to deliver Netflix traffic to their end users?

Netflix in the Nordics We provide our streaming service to Denmark Finland Norway Sweden Iceland and Greenland Currently co-located at Telecity Stockholm 1 & 2 Stockholm 1 is a network extension from Stockholm 2 due to space/power constraints Likely to expand to a second market in the Nordics this year

IXes Netflix delivers their Nordic traffic from four different IXes NetNod NIX AMS-IX LINX The latter two may not seem relevant here, but many networks have significant capacity built there

Standard build - European A European Netflix Open Connect IX build has: 20 Netflix Open Connect Appliances Two Juniper MX480 routers OpenGear console server and Juniper EX2200’s for OOB access Prewired MTP to LC cabling from Elpeus A “network extension” Open Connect build has: Two Juniper EX4550 switches Two MRV 40 channel passive muxes OpenGear console server for OOB access

Stockholm

Network Extension POP

Standard Build - US A US Netflix Open Connect IX build has: 40 Netflix Open Connect Appliances 60 Netflix Open Connect Flash Appliances Two Juniper MX960 routers OpenGear console server and Juniper EX2200’s for OOB access Prewired MTP to LC cabling from Elpeus A “network extension” Open Connect build has: Two Juniper EX4550 switches Two MRV 40 channel passive muxes OpenGear console server for OOB access

Converting some POPs to Cisco ASR Due to concerns with Juniper code delivery and the need for higher density platforms, we decided in early 2013 to integrate some Cisco ASR 9K into our network to replace Juniper MX In order to deliver our traffic goals (~1T per router), we need denser cards than the currently shipping Juniper 16 port MPC3’s

High Density… Juniper MX960 w/ MPC3’s can deliver 176 ports per chassis A single fabric failure brings the chassis to 3:2 oversubscription Juniper MX960 w/ MPC4’s can deliver 336 ports per chassis (oversubscribed) A single fabric failure (out of 3) brings the chassis to 2:1 oversubscription – unacceptable Cisco ASR9010 w/ Typhoon-based 36 port cards can deliver 288 ports per chassis

High Density, continued… We conservatively need 130 host ports per router Those 130 ports can deliver 1.13T of traffic Assuming we deliver 1 bit outbound for every bit inbound from a host, that puts us at 1.1:1 oversubscribed at peak 3:2 and 2:1 just do not work for us Juniper MPC4’s also require brand new 12.3 code What version of code are you running?

Cabling a new ASR…

The Netflix Open Connect Appliance (OCA) Developed in response to ISP requests to help scale Netflix traffic efficiently Reduces ISP cost by serving Netflix traffic from the local ISP datacenter, CO or headend, rather than upstream network interconnects Speeds up internet access for consumers to all third- party internet sites, because Netflix traffic is no longer a source of middle-mile or backbone congestion Netflix bears the capital and maintenance costs, not ISP ISP provides space, power and a network port An OCA is a component of the Netflix CDN (vs a cache)

OCA Hardware Space optimized: 4U high-density storage Power optimized for low power/cooling requirements (≅500W) Redundant 10GE optical network interfaces Redundant power supplies (AC or DC) (see slide)

Why Deploy a Netflix OCA? Netflix data is a significant percentage of ISP traffic in the markets we serve Serve >80% of Netflix traffic from the local ISP datacenter Remaining traffic can be served by upstream OCA’s, peering Saves on transit, transport and other upstream scaling costs Provided free of charge to participating ISPs ISPs with > 3 Gbps of Netflix traffic ISP provides rack space, power, 10 Gbps optical port(s)

OCA Operation Used exclusively for Netflix content Completely integrated with the Netflix content delivery system Greater effectiveness than transparent or proxy caches (>80% with Netflix OCA, versus 20-50% with other caches) Efficient content fill mechanisms Outside peak times (ISP selected time and BW per OCA) OCA offline during fill, staggered fill recommended. Fill can source from neighboring, peer or transit OCA’s ~5TB fill per day (i.e ~4Gb/s for ~2.5 hrs) OCA must be reachable by end users (port 80)

Directing Clients to OCAs Netflix Control Servers Broadband ISP User routing is done by Netflix control servers, not dependent on client DNS configuration Request is routed to the nearest available OCA Working sets of popular content deliver up to >80% network offload 2. Netflix control servers respond with address of local OCA 1. Client contacts Netflix control servers Netflix OCA 3. Client connects to local OCA 4. Local OCA delivers video stream

Network Positioning OCA’s are installed close to aggregations of end users Netflix works with ISPs to determine proper location and user-to-OCA mapping ISP Controls Traffic Flow End-user netblocks are associated with OCA via BGP advertisement Netflix honors AS path-length and MEDs allowing ISP to configure OCA hierarchy and failover OCA’s only serve content to netblocks provided by ISP ISPs choose peak throughput each OCA is allowed to generate

The Netflix ISP Speed Index Published monthly Helps consumers make an intelligent choice about their broadband provider We want our users to have the best experience possible We also want to help networks deliver Netflix traffic as efficiently as possible

A recently enabled network (night of 18/03/13) No marked increase in bandwidth utilization, but a more stable experience for users

Questions? Additional information: http://openconnect.netflix.com