Content Peering William B. Norton Co-Founder and Chief Technical Liaison Content Peering Forum September 18, 2002 © Equinix 2002

Internet Researcher White Papers: Focused studies on Peering 1.Identify Relevant Internet Operations Topic 2.Speak with prominent Peering Coordinators 3.Write/Evolve Draft White Paper 4.Walk Peering Coordinators through paper 5.Goto Step 3 Most widely read WP: “Internet Service Providers and Peering”

Peering Research Two-Year Study w/~200 Peering Coordinators –What is Peering? –How do you determine who to peer with? –When does Peering make sense? –When does Peering NOT make sense? –What is the Process of Peering? –How is Peering implemented? Goal: Document Internet Operations Practices  I will share with you today The Motives, The Methods, The Mindset of the Peering Coordinator Summary Findings of Peering Research….

Peering is a game of relationships

Level Setting Talk… Content Peering is… –Content-Heavy Companies leveraging Simple Internet Peering, Peering as ISPs do. –A Routing Optimization (direct vs. through an Intermediary) Not… –Content Peering Alliance (see –Or any end-to-end Content Alliance To this end I will share…

Agenda 1.Definitions Transit & Peering Terminology and Financial Cost Models 2.Introduction to the Peering Breakeven Analysis Specific Examples (San Jose Peering) leading to Generalizations 3.Peering Process 3 stages of Peering 4.Top 5 Reasons Not to Peer 5.Peering Resources Available to you Peering Makes Sense for Large Scale Content Players WP: “Internet Service Providers and Peering”

Definitions Gives us tools (lexicon) to facilitate discussion Market Confusion over misuse of terms: Terminology critical to have meaningful discussion on Peering Transit Peering Transport Tier 1 ISP Effective Peering Bandwidth Defs: ISPs

Understand that… 1.Def: The Internet is a network of networks 2.Def: Internet Service Providers sell access to the Internet 3.Internet Service Providers must themselves connect to the Internet. How do they connect to the rest of the Internet?

Definition of Transit 1) ISP A buys Transit Service Upstream Transit Provider ISP A Upstream Transit Provider Upstream Transit Provider Upstream Transit Providers Def: Transit is the business relationship whereby one ISP announces (usually sells) reachability to the *entire* Internet to a customer. $$$ ? Typically usage-based Pricing ’01: $100-$1200/Mbps Volume based on 95 th Percentile measure Transit is Simple, Convenient: Upstream handles the delivery of packets to the Internet by some means Usage: “I’m purchasing transit from Level 3.” 3) Traffic Flows INTERNETNETWORKSINTERNETNETWORKS 2) Reachability Announcement

Cost of Transit Traffic Exchange 2001 Pricing Sampling Range: $100-$1200/Mbps My Financial Models used $388/Mbps Source: 2002 Survey Range: $50-$1200/Mbps 95 th Percentile Blended Avg: ~$200/Mbps wholesale Transit Isn’t Transit Good Enough?

Traffic Growth Content Heavy ISP Access Heavy ISP UPSTREAMSUPSTREAMS V IG M B T Streaming Media 56k 384k 1.5m $$$ Yahoo! Broadcast: 100,000+ concurrent unicast streams 15 million streaming hrs/mo, 1300 Live events/day Access AOL+ DSL: 1,000/day Roadrunner Cable Modems: 1M subscribers KaZaa!!!! Two main motivations for Peering… Streaming: Broadcast Telephony Video Multimedia Interactive Gaming SPOT Events

Seek transport Interconnection $ x Why Peer? 2 Motivations for Peering 1.Financial : Reduce load on expensive Transit service Traffic src/dest Measure vs Intuit Usage-based Billing 2.Engineering : Lower latency 1 st Stage of Peering:  Top 10 destination ISP list Transit $$$ Transit $$$ ISP A ISP B Transit ISP Def: Peering…

Definition of Peering Def: Peering is the business relationship whereby ISPs reciprocally announce reachability to each others’ transit customers. Peering USNetEastNetWestNet Peering Transit Routing Tables Peering is *not* a transitive relationship Peering *does not* provide access to the entire Internet Cost of Peering: Def. Transport+Port Usage: “I buy transit from UUNet and Peer with EastNet, WestNet.”

The Costs of Peering Observation: Transport Prices have dropped like a rock. Observation: New Router prices have dropped like a rock. Observation: Used Router Market is also very healthy (cheap). Graphically…

Ethernet-Based Peering Model When does Peering make sense? (SJ Mkt) Flat Monthly Feesvs. Metered Monthly Fee

San Jose Market Prices for Peering Transit ISP ISP A ISP B Transit $$$ 1)Transport into Exchange 2) Rack Space at Exchange Point For Router ½ rack $2500/mo 3) Switch Port on Public Peering Fabric GigE R R X Peering $ Total Cost of Peering $7000/month Which is more cost effective? Peering or Transit? Allocate… 4) Cisco 7500 $2000/mo

Cost of Traffic Exchange Peering vs. solely Transit Transit ISP ISP A ISP B R R X Peering $ Cost of Peering: $7000/month 1 Mbps Unit Cost of Traffic Exchange In Peering Relationship: $7000=$7000 per Mbps 1 Mbps TRANSIT ~ $200/Mbps

Cost of Traffic Exchange Peering vs. solely Transit Transit ISP ISP A ISP B R R X Peering $ Cost of Peering: $7000/month 2 Mbps Unit Cost of Traffic Exchange In Peering Relationship: $7000=$3500 per Mbps 2 Mbps TRANSIT ~ $200/Mbps

Cost of Traffic Exchange Peering vs. solely Transit Transit ISP ISP A ISP B R R X Peering $ Cost of Peering: $7000/month 7 Mbps Unit Cost of Traffic Exchange In Peering Relationship: $7000=$1000 per Mbps 7 Mbps TRANSIT ~ $200/Mbps

Cost of Traffic Exchange Peering vs. solely Transit Transit ISP ISP A ISP B R R X Peering $ Cost of Peering: $7000/month 14 Mbps Unit Cost of Traffic Exchange In Peering Relationship: $4000=$500 per Mbps 14 Mbps TRANSIT ~ $200/Mbps

Cost of Traffic Exchange Peering vs. solely Transit Transit ISP ISP A ISP B R R X Peering $ Cost of Peering: $7000/month 70 Mbps Unit Cost of Traffic Exchange In Peering Relationship: $7000=$100 per Mbps 70 Mbps TRANSIT ~ $200/Mbps

OC-3 Peering vs. Transit in San Jose Generalized…

OC-12 Peering vs. Transit in San Jose Generalized… $15/Mbps

Peering Analysis Graph (axis)

Peering Analysis Graph (EPB) Definition: The Effective Peering Bandwidth is the maximum bandwidth available for peering, defined as the minimum of the available transport bandwidth and the usable bandwidth on the shared peering fabric.

Peering Analysis Graph (minCost) Definition: The Minimum Cost of Traffic Exchange is the unit cost of traffic exchange when the Effective Peering Bandwidth is fully utilized.

Peering Analysis Graph (EPR) Definition: The Effective Peering Range (EPR) is the range in which peering at an Internet Exchange makes sense (financially), measured as the range between the Peering Breakeven Point and the Effective Peering Bandwidth.

When does Peering Make Sense?

The 3 Stages of Peering

Interviews with 200 ISP Peering Coordinators revealed… 3 General Phases of Peering: 1) Identification of Potential Peer – the who 2) Initial Contact and Qualification – the why 3) Implementation Discussions – the how

I. Phase 1: Identification of Peer: Traffic Engineering Data Collection and Analysis Motivations: Reduce load on expensive Transit service Traffic src/dest Measure or Intuit Usage-based Billing 2 nd Goal: Lower latency Result  Top 10 list (see next page) Part of larger business deal Seek interconnection Transit $$ Transit $$ ISP A ISP B Transit ISP

Sample Top 10 Destination List

Phase 1: Identification of Potential Peer

II. Phase 2: Contact & Qualification, Initial Peering Discussion How to make contact with potential peer ISP? 1. person or 2.Exchange point participant list 3.Tech-c/admin-c from DNS/ASN registries 4.Engineering Forums NANOG, IETF, RIPE, etc. 5.Trade shows: speakers and booth staff 6.Target ISP sales force 7.Target ISP NOC

II. Phase 2: Contact & Qualification, Initial Peering Discussion Once contact is made… 1.Sometimes Mutual NDA 2.Exchange BiLateral Peering Agreement (BLPA) 3.Traffic Data justification shared One basis: Peering iff PeeringCost < TransitSavings? 4.Requirements Exchange (e.g. Must be at n Public Peering Points, xMbps, private peering migration strategy, etc.) Either Party may walk away….. If still interested, implementation discussion…

Phase 2: Contact and Qualification

III. Phase 3: Implementation Discussions How to interconnect? Direct Circuit-based Interconnection VS. Exchange-Based Interconnection

Cost Comparison at n=5 costDCfn()=(n-1)*C/2 $2,500 n=5 costDC=(4)*$2500/2 costDC=$5,000/mo costExchfn()=BDC+(n-1)*x/2+Racks $5,000 n=5, 1 costExch=$5,000+(4)(250/2)+$1000 costExch=$6,500/mo More expensive to use Exchange-Based Interconnection Strategy at n=5. N>5?

Exchange-based vs. Direct Circuit Interconnection Direct Circuits Model MUX Big Pipes Model Dark Fiber Model

9 Exchange Selection Criteria 1.Telecommunications Access Issues 2.Deployment Issues (getting in & up) 3.ISP Current Presences (there yet?) 4.Operations Issues (restrictions?) 5.Business Issues (neutrality/alignment) 6.Cost Issues ($$) 7.Credibility Issue (backing,attraction)  8.Exchange Population (side effect) 9.Existing vs. Emerging Exchange?

Value of the Internet Exchange V Exchange N Participants Cost Of Coming In (Circuits+ Routers+ StaffTime) V Capacity The Exchange Startup Hump First Carrier(s) First ISP(s) First CP(s) Critical Mass Point (V exchange =Cost Exchange ) Large Facility Scaling

Top 5 Reasons NOT to Peer

Top 5 Reasons not to Peer 1) Already get Traffic for “free” (through existing peering relationships) Transit $$$ Yahoo! Transit ISP EXODUS Peering $ AOL

Top 5 Reasons not to Peer 2) Not True Peers Traffic inequity Scale inequity Not even investments in infrastructure Form: “I don’t want to haul your traffic around the globe” Large Global Network Provider Small Regional Player Huge investment in Int’s circuits, 100’s of routers and colo sites, Staff installs, peering negotiations, Millions of customers, etc.

Top 5 Reasons Not to Peer 3) Lack of Technical Competence Troubleshooting network problems takes longer when the other ISP NOC and engineers lack the technical expertise during an outage…

Top 5 Reasons Not to Peer 4) Transit Sales Preferred We rather sell you transit…“Let me introduce you to our sales guys”

Top 5 Reasons Not to Peer 5) BGP is Tough “BGP? No Expertise  No measurements  No Justification to hire experts  BGP?” Primary Backup Transit Primary Backup Transit Conceptual Hurdle ComplexSimple 6: personality

Top 5 Reasons Not To Peer 5+ Personality Clashes: They don’t understand each other and they didn’t like the interaction

Resources Available to Peering Coordinators

Resources Available for Peering Coordinators Gigabit Peering Forums Other White Papers document Peering Practices Peering Contact Database

Gigabit Peering Forums

Other White Papers “Interconnection Strategies for ISPs” “Internet Service Providers and Peering” “A Business Case for Peering” “The Art of Peering: The Peering Playbook” “Do ATM-based Internet Exchange Points make sense anymore?” “The Peering Simulation Game” Freely available from the author:

Peering Contact Database For Peering Coordinators Only Toss in your Business Card & Receive a copy of everyone’s Business Cards Every 6 weeks (or so) Managed as a community service. to (Or give me your business card)

Conclusions Language of the Peering Coordinator –Transit, Peering, Transport ISP Peering makes sense if you can offload x Mbps of traffic to peers… The Peering Process includes 3 Phases: –Identification of Peer –Contact and Qualification –Implementation of Peering These represent the Baseline Understandings of the Peering Coordinator Acknowledgements?

Acknowledgements For this white paper I’d like to thank a few folks in particular for their review, insights, and comments on this paper: Dorian Kim (NTT/Verio), Ingrid Erkman (ICG), Dave McGaugh (ELI), Eric T. Bell (Time Warner Telecom), Chris Parker (StarNet), Brokaw Price (Yahoo!), Lane Patterson (Equinix), Jay Adelson (Equinix), Morgan Snyder (Equinix), John Hardie (Equinix), David Diaz (BellSouth), Joe Wood (Accretive Networks), Robert Seastrom (inter.net), Kevin Epperson (Level3), Petri Helenius (FICIX), Scott Sheppard (BellSouth), Ralph Doncaster (iStop.com), Leo Bicknell (ufp.org), Paul Vixie (vix.com), Ian Somerton and Dave Wodelet (Shaw/BigPipe), Tony Hain (Cisco), Jeff S. Wheeler (five-elements.com), Cliff Hafen, Dory Liefer, Shannon Lake (Omnivergent), Nenad Trifunovic (WorldCom), Andre Gironda (eBay), Jeb Linton (EarthLink), Daniel Golding (SockEye), Peter Moyer (Juniper), and others that preferred no recognition for their contributions to this paper. Top 5 Reasons Not To Peer…

Questions? Do these same Motivations to Peer apply to Content Companies? Are the Financial Motivations more compelling than the Performance Improvement Motivations?