1. Build vs. Buy: Internet Datacenter Operations Research INTAP March 6, 2001 Tokyo

2. Agenda My Background and Who is Equinix? – 15 minutes Recent Internet Operations Research – 20 minutes “Build vs.Buy Internet Datacenter” White Paper – 40 minutes Future: Where our Industry is heading – 5 minutes

3. NSFNET to Post-NSFNET Era Internet MY JOB: 5 Years as Chair of NANOG 12 Years in Internet…No Internet Preso… BARRNet NWNet Sesqui NEARNet NyserNet Merit NCAR OarNet Regional NSFNET SDSC AT&T MCI Sprint UUNet ANS NWNet NyserNet Merit OarNet Regional NSFNET (1987-1994) Regional Tech Meetings –Performance & Operations –Reporting & Coordination –Routing & Network Planning Privatization (1994-Now) North American Network Operators Group (NANOG) Meetings

4. Internet Statistics 87.5% of all Internet statistics are made up. Source: I made it up Source: I made it up.

5. Equinix, Inc. Scale – size of Internet Business Exchanges, how many, where they are, future plans “Virtual Fly Through” Tour of the IBX - 6 minutes

6. IBX™ Centers Industry Leading Security 5 security checkpoints to customer equipment State of the art video imaging and archiving On site security personnel 24x365 5 security checkpoints to customer equipment State of the art video imaging and archiving On site security personnel 24x365 Built to 3x power capacity 3 levels of power backup Forward looking design for growing industry Built to 3x power capacity 3 levels of power backup Forward looking design for growing industry Fully Redundant Backup Power 24x365 availability Customer trouble- shooting Smart Hands 24x365 availability Customer trouble- shooting Smart Hands IP/Carrier Knowledgeable Staff Customer Friendly Spaces – Conducting Business & Equipment Preparation Customer Friendly Spaces – Conducting Business & Equipment Preparation Designed from customer experience Ergonomic work spaces Designed from customer experience Ergonomic work spaces

7. IBX™ Growth Sites currently open or Under Development

8. Equinix Internet Business Exchange Model Carriers Internet Service Providers Content Providers Neutral Internet Business Exchange Model Infrastructure Players Neutrality…

9. Carrier Internet Exchange Model Internet Service Providers Carrier Operated Internet Exchange Infrastructure Players Non Carrier-Neutral No Content Support Economic Distortions Carrier Operator

10. Job Title: Equinix Co-Founder & Chief Technical Liaison So, my job is to take friends out for lunches and dinners… Internet Operations Research & Write White Papers Summary: What did I learn?

11. No Internet Operations Documentation To make things more difficult, subtle language differences!

12. The Internet Operators: All about relationships

13. My Job: White Papers Document Internet Operations Research –“Interconnection Strategies for ISPs” –“Internet Service Providers and Peering” –“Peering Case Studies” –“Peering Simulation Game” –“Build vs. Buy Internet Datacenter”  today New Job Title: Equinix Co-Founder & Chief Technical Liaison

14. White Paper: Interconnection Strategies for ISPs Direct circuits replaced with: OC-12 into Exchange Fiber cross connects Cost Drivers: Big OC-12 Direct Circuit Cheap Fiber Cross Connects Aggregation Efficiency over OC-12 (2:1,3:1) Rack Fees Exchange Model Cost Formula: ExchangeModelCost=BigDirectCircuit()+(n-1)*x/2+RackFees() So does the Exchange-based model make sense? OC-12  $23,000/mo XC=$200/mo Rack=$1500/mo

15. White Paper: Internet Service Providers & Peering Transit $$ Transit $$ ISP A ISP B Transit ISP Peering USNetEastNetWestNet Peering Routing Table

16. White Paper: Case Study: Peering in the Bay Area

17. White Paper: The Peering Simulation Game

18. White Paper: Build Vs. Buy Internet DataCenter Modeling: When does it make sense to build your own? To Outsource? Issues to consider Economies of Scale? Multiple Business Models

19. Service Level Agreements in Principle Guarantee a higher level quality Define Measurement Criteria for adhering to higher quality service Level Meeting Performance Level critical to success of business Failure to meet Service Levels penalty significantly worse than cost of service –eBay: $64,000/minute of downtime

20. SLAs in Practice No SLA Pricing –ISPPrice NO SLA =X SLA Pricing –ProbabilityOfMissingSLA=p –ISPPrice SLA1 =X*(1+p) Network does not typically change to meet new criteria CIO Security Audit Story Equinix Switch SLA: 99.999% Uptime

21. SLA-Datacenters State-of-the-art Datacenters Infrastructure Failure unlikely SLAs rare, but… Typically Limited to free month service (Insufficient to potential loss customer faces) But what is an Internet Datacenter? Include hosting? Include Internet Service? Include Caching, Content Distribution? Models help Define and Frame the Problem…

22. SLA Datacenter Parameters If there were Internet Datacenter SLAs Parameters: –Installation Lead Time –Service Availability –24/7 Customer Support –Provisioning/Responsiveness Independent of Transport and Internet Service Issues.

23. Creating the Build vs. Buy Internet Datacenter Whitepaper Create a community white paper on issues associated with the building & scaling of Internet Datacenters Identify the issues that folks should consider when making the build vs. buy decision Today: Demonstrate the document & models to date and get feedback NOTE WELL: EARLY STAGE OF BVB WHITE PAPER (v0.4)! This Model may be of help to you in your SLA Discussions

24. Toured 32 (now) data centers

25. Some with very Secure Fiber Access… Source: Technology Center of the Americas Tour

26. Some with less secure fiber entry… Source: Don’t Ask.

27. Some Very Secure… http://www.thebunker.net

28. Some Cat 5 Hurricane proof…

29. Some not quite cat proof. Loading Dock

30. Great religious debate over ties Velcro vs. Plastic cable ties vs. String

31. Time to white paper what I’ve seen & Facilitate BOF Discussion Datacenter BOF I NANOG Oct ’98 Atlanta Build vs. Buy White Paper Internet Datacenter Model –Definition of Internet Datacenter Focused discussion of Each Component of the Model

32. Internet Datacenter Model Dependence Relationship Increased Value

33. Dependence Customer Business Model

34. Value Dependence Internet Datacenter Model Example IXP

35. Ultimate Goal Build Financial models that allow an ISP to say: “Given our size requirements, it makes sense to build our own.” Or “Given our size requirements, it probably makes sense to outsource this to someone who can realize economies of scale.” + + = Buy Build Size Unit cost

36. Flavor of the Real Estate Model Place to put stuff Proximity to Fiber Space Size Utility Power Facilities Yard Increased Value Relationship Scaling of Datacenter Real Estate?

37. Scaling of the Real Estate Model # Sq. Ft. Premium Real Estate Market (Tokyo, NYC,…) Real Estate Market (everywhere else) $/sq. ft.

38. Real Estate Size is Important Choose Too Small: Run out of space, lose economies of scale Choose Too Big: Cost per Rack very high Vs. Right of First Refusal (some Internet Datacenters allow this) Expandability of Space

39. Commitment to DataCenter 1 yr contract Vs. 10 year lease Time to Market Issue – 6 – 9 months to build

40. Hidden Charges Meet Me Rooms Required products: –Must buy port on switch or certain volume of BW Cross Connect Fees

41. Size & Finding Real Estate 100,000 sq ft in downtown DC? 10,000 sq ft in downtown DC Vs. Fiber Miles to Internet Datacenter

42. Flavor of the Infrastructure Model Heating Venting Air Conditioning (HVAC) Security Infrastructure Power Infrastructure Fire Suppression

43. Heating, Venting, Air Conditioning (HVAC) Infrastructure Controlled Humidity ~45% Too much Humidity: System Failure Too Little Humidity: Static Electricity Air Temperature:~72 (IBM: 65-85) Solutions: DX Systems Air Cooled Water-Based How do these Scale?

44. HVAC Scaling Direct Expansion Air Cooled Chilled Water System HVAC Cost # Racks $1,000/ton $478/ton $250/ton HVAC System Cost: Economies of Scale DX Systems: 25-300 tons of Cooling ~ 10K sq ft or ~150-200 racks Terramark Outsource of HVAC to FPL…

45. Crossover point where Chilled Water lowest per-rack solution

46. Trend? Outsourcing of HVAC FPL HVAC (Miami FL) Water Chiller NOTA: 3 Blocks Away Source: NAP of the Americas Tour

47. Security Infrastructure 24x7 Manned Access: Biometrics, card keys Video Surveillance Sliding Glass

48. Power Infrastructure Sun Netra 1U=90Watts –40U Rack=3600 Watts! Power Source –Power Grid (dual feeds) –Co-Generation –UPS/CPS/Generators Energy Storage –UPS Batteries –Flywheel (no UPS) –Fuel Cells (service) Monitoring Issues Redundancy Issues TBD: Scaling Issues

49. Fuel Cells for Generators

50. Fire Suppression Infrastructure Two Dominant Systems: VESPA air sampling Dry pipe, pre action water system vs. FM-200 Smoke & Heat Required to activate region

51. Infrastructure Operations Operations of HVAC, Security, Power systems N+1 Redundancy

52. Internet Datacenter Model

53. Transport Service Model Dark Fiber Provider(s), Carrier(s), Aggregators (like Universal Access), etc. Business Model? –Owned by Carrier? Will others come in? –Not owned by Carrier? Will Carriers come in? What type of services available? How long to Provision? Big Problem: Keeping up with BW Demand

54. Internet Datacenter Model

55. Internet Service Model ISP(s), Aggregator(s) like InterNAP? Business Model –Owned by ISP? Will others come in? –Not owned by ISP? Will others come in? Capacity Issues Latency Issues Interconnection Strategies Allowed?

56. Internet Datacenter Model

57. Content Provider Model Hoster Focus: –Installing New Servers, Provisioning Host Services, –Managing Server Farm, SLAs, Tech Support, 24/7 NOC, Back Office Solutions Hoster owns Datacenter? –Pro: Can buy Transit Services –Con: Expensive w/Bandwidth Hungry Apps –Con: Shared bandwidth issues (Yahoo! attack) –Con: Local Loop Expensive Business Model Content Distributor(s), Hoster(s)

58. Internet Datacenter Model

59. Future of the IXP? MarketPlace: Stratification of component operations, fast creation & provisioning of packaged operations services Implication: IXP  Content Sources,  bandwidth demands DSL Access Providers Dial-in Modem Operators E-Mail Svcs USENET Svcs WebHost Svcs Search Engines Cache/Content Distribution Carrier Internet Service Provider Carriers Carrier Internet Service Provider Carrier Internet Service Provider Service Providers Internet Service Provider Wireless Access Svc Virtual ISP Sparing

60. Future SLA Challenges 1.Bundling of Internet Operations Services… 2.Into Cohesive SLA-able services 3.Notion: Trickle-down SLA Bundle Participant Operations Agreements Financial Agreements Example: Bundle Internet DataCenter, Transport, Internet, MSP. What happens when Transport service fails? Who covers the cost of the SLA?

61. BvB Document Availability All references are in the BvB white paper –Hyperlinks work in electronic version of document –Early Stage of document: Comments Welcome! Send e-mail to wbn@equinix.com Subject: Build vs. Buy White Paperwbn@equinix.com

62. Thanks to Sean Donelan (Equinix) Jason Colangelo (APC) Stephen Stuart/Phil Steffora (MFN) All Colo/ISPs tour facilitators Jay Adelson (Equinix) Josh Horn (NOTA) Justin Newton (NetZero) Friends at Exodus, Frontier Global Center, Level3, etc. Daniel Golding (NetRail) Kenney Huang (AsiaInfra) Sandip Gupta (Ensim) Rob Glenn (Sun) Everyone here! Please e- mail feedback.