Build vs. Buy: Internet Datacenter Operations Research INTAP March 6, 2001 Tokyo

Agenda My Background and Who is Equinix? – 15 minutes Recent Internet Operations Research – 20 minutes “Build vs.Buy Internet Datacenter” White Paper – 40 minutes Future: Where our Industry is heading – 5 minutes

NSFNET to Post-NSFNET Era Internet MY JOB: 5 Years as Chair of NANOG 12 Years in Internet…No Internet Preso… BARRNet NWNet Sesqui NEARNet NyserNet Merit NCAR OarNet Regional NSFNET SDSC AT&T MCI Sprint UUNet ANS NWNet NyserNet Merit OarNet Regional NSFNET ( ) Regional Tech Meetings –Performance & Operations –Reporting & Coordination –Routing & Network Planning Privatization (1994-Now) North American Network Operators Group (NANOG) Meetings

Internet Statistics 87.5% of all Internet statistics are made up. Source: I made it up Source: I made it up.

Equinix, Inc. Scale – size of Internet Business Exchanges, how many, where they are, future plans “Virtual Fly Through” Tour of the IBX - 6 minutes

IBX™ Centers Industry Leading Security 5 security checkpoints to customer equipment State of the art video imaging and archiving On site security personnel 24x365 5 security checkpoints to customer equipment State of the art video imaging and archiving On site security personnel 24x365 Built to 3x power capacity 3 levels of power backup Forward looking design for growing industry Built to 3x power capacity 3 levels of power backup Forward looking design for growing industry Fully Redundant Backup Power 24x365 availability Customer trouble- shooting Smart Hands 24x365 availability Customer trouble- shooting Smart Hands IP/Carrier Knowledgeable Staff Customer Friendly Spaces – Conducting Business & Equipment Preparation Customer Friendly Spaces – Conducting Business & Equipment Preparation Designed from customer experience Ergonomic work spaces Designed from customer experience Ergonomic work spaces

IBX™ Growth Sites currently open or Under Development

Equinix Internet Business Exchange Model Carriers Internet Service Providers Content Providers Neutral Internet Business Exchange Model Infrastructure Players Neutrality…

Carrier Internet Exchange Model Internet Service Providers Carrier Operated Internet Exchange Infrastructure Players Non Carrier-Neutral No Content Support Economic Distortions Carrier Operator

Job Title: Equinix Co-Founder & Chief Technical Liaison So, my job is to take friends out for lunches and dinners… Internet Operations Research & Write White Papers Summary: What did I learn?

No Internet Operations Documentation To make things more difficult, subtle language differences!

The Internet Operators: All about relationships

My Job: White Papers Document Internet Operations Research –“Interconnection Strategies for ISPs” –“Internet Service Providers and Peering” –“Peering Case Studies” –“Peering Simulation Game” –“Build vs. Buy Internet Datacenter”  today New Job Title: Equinix Co-Founder & Chief Technical Liaison

White Paper: Interconnection Strategies for ISPs Direct circuits replaced with: OC-12 into Exchange Fiber cross connects Cost Drivers: Big OC-12 Direct Circuit Cheap Fiber Cross Connects Aggregation Efficiency over OC-12 (2:1,3:1) Rack Fees Exchange Model Cost Formula: ExchangeModelCost=BigDirectCircuit()+(n-1)*x/2+RackFees() So does the Exchange-based model make sense? OC-12  $23,000/mo XC=$200/mo Rack=$1500/mo

White Paper: Internet Service Providers & Peering Transit $$ Transit $$ ISP A ISP B Transit ISP Peering USNetEastNetWestNet Peering Routing Table

White Paper: Case Study: Peering in the Bay Area

White Paper: The Peering Simulation Game

White Paper: Build Vs. Buy Internet DataCenter Modeling: When does it make sense to build your own? To Outsource? Issues to consider Economies of Scale? Multiple Business Models

Service Level Agreements in Principle Guarantee a higher level quality Define Measurement Criteria for adhering to higher quality service Level Meeting Performance Level critical to success of business Failure to meet Service Levels penalty significantly worse than cost of service –eBay: $64,000/minute of downtime

SLAs in Practice No SLA Pricing –ISPPrice NO SLA =X SLA Pricing –ProbabilityOfMissingSLA=p –ISPPrice SLA1 =X*(1+p) Network does not typically change to meet new criteria CIO Security Audit Story Equinix Switch SLA: % Uptime

SLA-Datacenters State-of-the-art Datacenters Infrastructure Failure unlikely SLAs rare, but… Typically Limited to free month service (Insufficient to potential loss customer faces) But what is an Internet Datacenter? Include hosting? Include Internet Service? Include Caching, Content Distribution? Models help Define and Frame the Problem…

SLA Datacenter Parameters If there were Internet Datacenter SLAs Parameters: –Installation Lead Time –Service Availability –24/7 Customer Support –Provisioning/Responsiveness Independent of Transport and Internet Service Issues.

Creating the Build vs. Buy Internet Datacenter Whitepaper Create a community white paper on issues associated with the building & scaling of Internet Datacenters Identify the issues that folks should consider when making the build vs. buy decision Today: Demonstrate the document & models to date and get feedback NOTE WELL: EARLY STAGE OF BVB WHITE PAPER (v0.4)! This Model may be of help to you in your SLA Discussions

Toured 32 (now) data centers

Some with very Secure Fiber Access… Source: Technology Center of the Americas Tour

Some with less secure fiber entry… Source: Don’t Ask.

Some Very Secure…

Some Cat 5 Hurricane proof…

Some not quite cat proof. Loading Dock

Great religious debate over ties Velcro vs. Plastic cable ties vs. String

Time to white paper what I’ve seen & Facilitate BOF Discussion Datacenter BOF I NANOG Oct ’98 Atlanta Build vs. Buy White Paper Internet Datacenter Model –Definition of Internet Datacenter Focused discussion of Each Component of the Model

Internet Datacenter Model Dependence Relationship Increased Value

Dependence Customer Business Model

Value Dependence Internet Datacenter Model Example IXP

Ultimate Goal Build Financial models that allow an ISP to say: “Given our size requirements, it makes sense to build our own.” Or “Given our size requirements, it probably makes sense to outsource this to someone who can realize economies of scale.” + + = Buy Build Size Unit cost

Flavor of the Real Estate Model Place to put stuff Proximity to Fiber Space Size Utility Power Facilities Yard Increased Value Relationship Scaling of Datacenter Real Estate?

Scaling of the Real Estate Model # Sq. Ft. Premium Real Estate Market (Tokyo, NYC,…) Real Estate Market (everywhere else) $/sq. ft.

Real Estate Size is Important Choose Too Small: Run out of space, lose economies of scale Choose Too Big: Cost per Rack very high Vs. Right of First Refusal (some Internet Datacenters allow this) Expandability of Space

Commitment to DataCenter 1 yr contract Vs. 10 year lease Time to Market Issue – 6 – 9 months to build

Hidden Charges Meet Me Rooms Required products: –Must buy port on switch or certain volume of BW Cross Connect Fees

Size & Finding Real Estate 100,000 sq ft in downtown DC? 10,000 sq ft in downtown DC Vs. Fiber Miles to Internet Datacenter

Flavor of the Infrastructure Model Heating Venting Air Conditioning (HVAC) Security Infrastructure Power Infrastructure Fire Suppression

Heating, Venting, Air Conditioning (HVAC) Infrastructure Controlled Humidity ~45% Too much Humidity: System Failure Too Little Humidity: Static Electricity Air Temperature:~72 (IBM: 65-85) Solutions: DX Systems Air Cooled Water-Based How do these Scale?

HVAC Scaling Direct Expansion Air Cooled Chilled Water System HVAC Cost # Racks $1,000/ton $478/ton $250/ton HVAC System Cost: Economies of Scale DX Systems: tons of Cooling ~ 10K sq ft or ~ racks Terramark Outsource of HVAC to FPL…

Crossover point where Chilled Water lowest per-rack solution

Trend? Outsourcing of HVAC FPL HVAC (Miami FL) Water Chiller NOTA: 3 Blocks Away Source: NAP of the Americas Tour

Security Infrastructure 24x7 Manned Access: Biometrics, card keys Video Surveillance Sliding Glass

Power Infrastructure Sun Netra 1U=90Watts –40U Rack=3600 Watts! Power Source –Power Grid (dual feeds) –Co-Generation –UPS/CPS/Generators Energy Storage –UPS Batteries –Flywheel (no UPS) –Fuel Cells (service) Monitoring Issues Redundancy Issues TBD: Scaling Issues

Fuel Cells for Generators

Fire Suppression Infrastructure Two Dominant Systems: VESPA air sampling Dry pipe, pre action water system vs. FM-200 Smoke & Heat Required to activate region

Infrastructure Operations Operations of HVAC, Security, Power systems N+1 Redundancy

Internet Datacenter Model

Transport Service Model Dark Fiber Provider(s), Carrier(s), Aggregators (like Universal Access), etc. Business Model? –Owned by Carrier? Will others come in? –Not owned by Carrier? Will Carriers come in? What type of services available? How long to Provision? Big Problem: Keeping up with BW Demand

Internet Datacenter Model

Internet Service Model ISP(s), Aggregator(s) like InterNAP? Business Model –Owned by ISP? Will others come in? –Not owned by ISP? Will others come in? Capacity Issues Latency Issues Interconnection Strategies Allowed?

Internet Datacenter Model

Content Provider Model Hoster Focus: –Installing New Servers, Provisioning Host Services, –Managing Server Farm, SLAs, Tech Support, 24/7 NOC, Back Office Solutions Hoster owns Datacenter? –Pro: Can buy Transit Services –Con: Expensive w/Bandwidth Hungry Apps –Con: Shared bandwidth issues (Yahoo! attack) –Con: Local Loop Expensive Business Model Content Distributor(s), Hoster(s)

Internet Datacenter Model

Future of the IXP? MarketPlace: Stratification of component operations, fast creation & provisioning of packaged operations services Implication: IXP  Content Sources,  bandwidth demands DSL Access Providers Dial-in Modem Operators Svcs USENET Svcs WebHost Svcs Search Engines Cache/Content Distribution Carrier Internet Service Provider Carriers Carrier Internet Service Provider Carrier Internet Service Provider Service Providers Internet Service Provider Wireless Access Svc Virtual ISP Sparing

Future SLA Challenges 1.Bundling of Internet Operations Services… 2.Into Cohesive SLA-able services 3.Notion: Trickle-down SLA Bundle Participant Operations Agreements Financial Agreements Example: Bundle Internet DataCenter, Transport, Internet, MSP. What happens when Transport service fails? Who covers the cost of the SLA?

BvB Document Availability All references are in the BvB white paper –Hyperlinks work in electronic version of document –Early Stage of document: Comments Welcome! Send to Subject: Build vs. Buy White

Thanks to Sean Donelan (Equinix) Jason Colangelo (APC) Stephen Stuart/Phil Steffora (MFN) All Colo/ISPs tour facilitators Jay Adelson (Equinix) Josh Horn (NOTA) Justin Newton (NetZero) Friends at Exodus, Frontier Global Center, Level3, etc. Daniel Golding (NetRail) Kenney Huang (AsiaInfra) Sandip Gupta (Ensim) Rob Glenn (Sun) Everyone here! Please e- mail feedback.