Optical Networks for the Rest of Us “Customer Empowered Networking” NANOG 17 – Montreal Background Papers on Gigabit to The Home and Optical Internet Architecture Design Available Optical Internet News list: Send to Tel:

 Mission: To facilitate the development of Canada’s communications infrastructure and stimulate next generation products, applications and services  Canadian equivalent to Internet 2 and NGI  private-sector led, not-for-profit consortium  consortium formed 1993  federal funding of $220m ( )  total project costs estimated over $500 M  currently over 140 members; 21 Board members  Phase III funding to be announced  $55 million announced for Optical Internet -March 1998 CANARIE Inc

GigaPOP CA*net 3 National Optical Internet Vancouver Calgary Regina Winnipeg Ottawa Montreal Toronto Halifax St. John’s Fredericton Charlottetown ORAN BCnet Netera SRnet MRnet ONet RISQ ACORN Chicago STAR TAP CA*net 3 Primary Route Seattle New York Los Angeles CA*net 3 Diverse Route Deploying a 4 channel CWDM Gigabit Ethernet network – 700 km Deploying a 4 channel Gigabit Ethernet transparent optical DWDM– 1500 km Multiple Customer Owned Dark Fiber Networks connecting universities and schools 16 channel DWDM -8 reserved for CANARIE -8 wavelengths for carrier and other customers Consortium Partners: Bell Nexxia Nortel Cisco JDS Uniphase Newbridge Condo Dark Fiber Networks connecting universities and schools

What is an Optical Internet?  WDM fibers where individual wavelengths are the link layer interconnect directly connected to routers via Optical ADM (Add Drop Mux) or WDM coupler  High Performance Router acts as the main switching routing device  Bypass or cut-thru connections via dedicated wavelengths  SONET or Gigabit Ethernet framing (also 10xGbE or SDL)  Use intrinsic self healing nature of Internet for redundancy and protection (don’t require SONET/SDH layer)  Traffic engineering and network management done via MPLS  Network design optimized for unique characteristics of Internet traffic – fractal traffic, asymmetric traffic and congestion at the edge

Lessons Learned  Carrier transport people now must learn to deal with customers directly  Require network management tools that give customer a view of “their” wavelengths  A whole new set of operating procedures required  OAM&P issues between router vendors and DWDM remain a challenge  SONET management systems expect to see a contiguous network  CA*net 3 required DCC work arounds  Need network tools to measure end to end performance and throughput at OC-48 or greater speeds –  HP is about to release a couple of beta products  MPLS is proving a lot more difficult in practice to implement  Need tools for management of tunnels  Need Inter-domain MPLS-TE  Mythology of 50msec “fast restoral” still not understood  OSPF with very short hold down timers and GRE tunnels or policy routing may be an adequate alternative

10xGbE & CWDM  Several companies have announced long haul GbE and CWDM with transceivers at 50km spacing  10GbE coming shortly  Costs are as little as $12K US per node (or transceiver)  Future versions will allow rate adaptive clocking for use with “gopher bait” fiber, auto discovery, CPE self manage  Excellent jitter specification  Most network management and signaling done at IP layer  Anybody with LAN experience can build a long haul WAN – all you need is dark fiber  With CWDM, no EDFA power disbursement  Repeater distance independent number of wavelengths

 Many ISPs, regional networks, municipalities, school districts are purchasing dark fiber or building dark fiber networks up to 1000 km rather than managed bandwidth  With dark fiber increased bandwidth only entails upgraded equipment costs and no additional monthly charges  Significant savings in relocating servers to central site and using VoIP  Also many carriers willing to sell “gopher bait” fiber (fiber that does not meet stringent SONET/DWDM requirements) at a discount  As such, cost of transmission equipment is becoming a significant factor versus cost of fiber  SONET and ATM networks require specialized engineering knowledge and skills  Customers want a technology in the WAN they are familiar with and that is easily extensible from the LAN e.g. Ethernet  Don’t require the same reliability as telco systems Market drivers for GbE in the WAN

Major Long Haul Components for IP/DWDM network SONET Regen $250k per Tx/Rx Approximate Distances for OC-192 system Typical Cost $6000 per km (not counting cost of fiber router, and transponder) for one OC-192 channel Advantage – can support multi-services and well known technology WDM Coupler $20K 250 km 50 km Wideband Optical Repeater $250K SONET Transport Terminal Transponder For transponder currently using regen box $125K Terabit Router $400K

Major Long Haul Components for 10xGbE CWDM network Approximate Distances for 10xGbE system Typical Cost $400 per km (not counting cost of fiber Or 10xGbE switches) for 10 Gbps Advantage – very low cost 1/10 cost of SONET & DWDM Disadvantage – requires 2 fibers and can only carry IP traffic CWDM Coupler $5K 50 km 10x Transceiver $20K 10xGbE Switch $20K GGGG

Optical Networks for the Rest of Us  With customer owned dark fiber, 10GbE and 4 channel CWDM anybody can build a 40Gbps network up to 1000km or greater at a fraction of cost of traditional telco network  May not be suitable for mission critical traffic (at least not yet)  But ideal for high bandwidth Internet to the school, small business and home  Ring structures are a customer option – not a mandatory requirement  The driver is NOT new applications, but cost savings –1 year payback  Typical cost is one time $20K US per school for a 20 year IRU  In Ottawa we are deploying a 60km- 96 strand network connecting 22 institutions – cost $500K US

Where are we going?  Today the Internet is “virtual” network riding on top of a traditional “connection oriented” network of cooper and fiber  With optical technology such as customer owned dark fiber, customer owned wavelengths, 10GbE etc we can extend the model of the Internet as tool to empower the user to build networks in the physical domain as well as the virtual domain  The future telecommunication’s world may be dominated by thousands of customer owned networks that peer at the physical as well as at the virtual level, “Optical Networks for the Rest of Us”  A national or provincial K-12 network with its own wavelengths and dark fiber  A national banking network with its own wavelengths and dark fiber  A national auto network with its own wavelengths and dark fiber  A radical departure from the “carrier centric” view of the universe

2 Different Views Telco Network “Ring of Rings” % reliability only in the SONET Ring for the telco, no guarantees for the customer ISP A ISP B ISP C CO Customer Empowered Network “Warp and Weave” Customer responsible for building rings % reliability to the customer but no guarantees for the ISP SONET ring 3 separate dark fiber builds

Customer Empowered Networks ISP A ISP B ISP C ISP A ISP B ISP C Dark fiber or CWDM Network City A First Dark fiber Network City B Dim Wavelength Long Haul DWDM Second Dark Fiber Network Customer achieves reliability by multi-home to different ISPs ISP D Optical Label Switched Router Dark fiber Network City C

New Challenges and Opportunities  “Customer empowered networks” present a whole new set of research challenges:  Peering and topology protocols in the optical domain – what will be the equivalent to BGP and OSPF in the optical domain  Multi Protocol Lambda Switching?  Defining LSP attributes such as power level, wavelength, encoding, etc?  Interdomain optical MPLS?  Customer controlled establishment of wavelengths, routing and service delivery  Auto discovery of wavelengths?  Management and interface systems, etc, etc