CANARIE CA*net 4 Design Document Last Revised April Version OBGP documentation and latest version of this document can be found at Tel:

Why CA*net 4  Traffic growth on CA*net 3:  Mbps average vs. Internet 2 traffic of 300 Mbps Average and peak > 1.5 Gbps  traffic doubling every 3 – 4 months  at current growth rates C3 will be congested 4Q2001  Many universities are still connected at low speeds e.g. most Ontario universities are only connected at 5 –10 Mbps  Many ORANs are just starting to get deployed  Expected growth in new high bandwidth applications that cannot be supported over the commercial Internet  Wavelength Disk Drive applications  Continuous HDTV broadcast from the space station – 1Q01  McGill University HDTV media wall – 2Q01  UoSaskatoon Synchrotron project – 4Q01  Gemini Digital Telescopes go on line – 2Q01  Number of extreme high bandwidth Grid projects planned – Neptune, Gryphen, Pacific Forestry, etc  Growing recognition that research community needs a permanent advanced network platform to support future research need

The Concept for CA*net 4  Conventional optical networks are built on the paradigm that a central entity has control and management of the wavelengths  It therefore must have control of the edge device for the setup and tear down of the wavelengths  Will central control and management scale to millions of edge device and thousands of optical wavelengths?  Customer empowered optical networks are built on the paradigm that customer owns and controls the wavelengths (Virtual Dark Fiber)  Customer controls the setup, tear down and routing of the wavelength between itself and other customers  Customer may trade and swap wavelengths with other like minded customers ultimately leading to wavelengths as market commodity  How do you design a network architecture if the routing and control of wavelengths is under the control of the customer at the edge?  Network is now an asset, rather than a service  Analogy to time sharing computing in the early 1970s versus customer owned computers or client-server computing

CA*net 4 & Community Networks  CA*net 4 will be a national resource for community networks and supporting NBTF initiatives  eScience grids, learning grids and health grids  Researchers & educator will want to use computing resources of schools and homes as part of large distributed computing projects  CA*net 4 will interconnect environmental and health grids with students and researchers  New grid projects in bio-informatics, pharmaceutical research, particle physics need access to millions of computers  Next big discovery in cancer or particle physics could be made at your local high school

Condo Fiber & Wavelengths  Condo fiber means that separate organizations own individual strands of fiber in a fiber cable  Each strand owner responsible for lighting up the strand  Collectively responsible for sharing costs of maintenance on fiber cable, relocation etc  Condo wavelengths  Number of parties share in the cost of a single strand and that light up with an agreed upon number of wavelengths  Wavelengths are portioned based on percentage ownership  With condo fiber and condo wavelengths institutions can treat network as an asset just like purchasing a computer, rather than a service as today

Research Network Issues  Research and Education networks must be at forefront of new network architecture and technologies  Should be undertaking network technology development that is well ahead of any commercial interest  But any network architecture can only be validated by connecting real users with real applications and must solve real world problems  Test networks per se are not sufficient  There is a growing trend for many schools, universities and businesses to control and manage their own dark fiber  Can we extend this concept so that they can also own and manage their own wavelengths?  Will “empowering” customers to control and manage their own networks result in new applications and services similar to how the PC empowered users to develop new computing applications?

CA*net 4 Research Objective  To deploy a network architecture where the GigaPOPs and institutions at the edge manage and control their own fiber and their own wavelengths  Condominium fiber and condominium wavelengths  To deploy a novel new optical network of distributed optical IXs that gives GigaPOPs and communities at the edge of the network (and ultimately their participating institutions) the ability to setup and manage their own wavelengths across the network and thus allow direct peering between GigaPOPs on dedicated wavelengths and optical cross connects that they control and manage  To allow the establishment of wavelengths by the GigaPOPs and their participating institutions in support of eScience and grid applications to support true peer-to-peer networking  To allow connected regional and community networks to setup peering relationships with CA*net 4 for collaborative research and education and eScience applications  To partner with private sector in building “carrier neutral” distributed optical Internet exchange facilities across Canada and developing new services in fungible wavelengths to enable customer empowered optical networks

Problems with Current Architecture  All networks today are hierarchical  Based on a network paradigm to support voice telephony where N is largely stable but ultimately limited by number of humans  On Internet N is only limited by number of computers and number of processes running on computers  N is unbounded and will continue to grow fast  Internet appliances, wireless, and P2P applications  Result is that core of Internet will have to grow faster than offered load  Large scale networks may not be sustainable  Two factors driving Internet growth:  Volume (Amplitude)  N or Topology (Phase)  Other factors compound the problem  Distance invariance, temporal self similarity (burstiness), spatial self similarity (slosh), etc

Current View of Optical Internets Big Carrier Optical Cloud using MP S or ASON for management of wavelengths for provisioning, restoral and protection Carrier controls and manages edge devices Optical VLAN Customer ISP AS 1 AS 2 AS 3 AS 1 AS 4 AS 5 UNI NNI

Customer Empowered Metro Network Carrier Neutral IX & OBGP switch City A City B City C Carrier Neutral IX & OBGP switch Condo Dark Fiber Condo Wavelengths OBGP switch

Future Optical Networks Customer A Customer B Customer C Customer D Customr A elects to cross connect with Customer C rather than D Massive peering at the edge Condo Fiber Condo Wavelength

CA*net 4 Research Areas  New optical technologies that support customer empower networking  OBGP, CWDM, hybrid optics and HWDM, customer controlled optical switches  BGP scaling issues  Object Oriented Networking  Wavelengths and optical switch treated as an object and method to be incorporate into middleware  Or treated as fungible product  Distributed Computing Applications and Grids  Wavelength Disk Drives (WDD)  eScience  Grids for weather forecasting, forestry management, education, health, etc

Object Oriented Networking  Combines concepts of Active Networks and Grids  See DARPA  See Globus  Customer owns sets of wavelengths and cross connects on an optical switch  Network elements can be treated as a set of objects in software applications or grids  Complete with inheritances and classes, etc  Rather than distributed network objects ( e.g. Java or Corba) distributed object networks  In future researchers will purchase networks just like super computers, telescopes or other big science equipment  Networks will be an asset – not a service  Will be able to trade swap and sell wavelengths and optical cross connects on commodity markets

Advantages of OON  With massive peerings to the edge, the loss of one peer is not catastrophic  No need for restoral or protection paths or ring architectures  Networks look more like “star bursts” rather than “ring of rings”  See C Labovitz ACM Sigcomm Aug 2000 – massive peering helps faster convergence  May solve problem of scaling large networks  Today M carriers building meshed networks to N customers with resultant M*N 2 requirement for wavelengths  With OON the global requirement for wavelengths grows at X*N where X= average number of wavelengths per customer

Example OON  Earthquake Visualization Grid  Globus Middleware Begin  Establish connection to other grid participants  Network Object – wavelength to STAR LIGHT – Chicago  Network Object – wavelength to Research center Amsterdam  Network Object – wavelength to SDSC Visualization Computer  Network Object – wavelength to Seismology Center Calgary  Link objects and create grid  Run Visualization  Release Network objects  Globus Middleware End  Earthquake Visulization End

Napster OON  University in Canada willing to exchange these wavelengths  Montreal-NY blue  NY-Amsterdam red  Chicago-Montreal green  Wants these wavelengths  Chicago-SDSC - purple  SDSC- Hawaii - red  Chicago- Chile - yellow  University in Chicago willing to swap these wavelengths  Chicago to SDSC – purple  SDSC- Hawaii – Red  Chicago to Argonne - Blue  Wants wavelength to these locations  Chicago to Toronto - yellow

 “National grand challenge" e-research projects are on the horizon: with the next generation network, interconnecting to school and community networks, Canadian researchers could use the thousands of computers in schools and communities distributed across Canada  Students at schools and ultimately members of the public could be full participants in basic reserach  The next generation research network should be designed to encourage and enable projects such as these The eScience Vision

Wavelength Disk Drives Vancouver Computer data continuously circulates around the WDD Calgary Regina Winnipeg Ottawa Montreal Toronto Halifax St. John’s Fredericton Charlottetown CA*net 3/4 WDD Node

eScience Grid Customer A Customer B Customer C Customer D WDD Grid Customers autonomously create WDD ring for high performance applications

Wavelength Disk Drives  CA*net 4 will be “nation wide” virtual disk drive for grid applications  Big challenges with grids or distributed computers is performance of sending data over the Internet  TCP performance problems  Congestion  Rather than networks being used for “communications” they will be a temporary storage device  Ideal for “processor stealing” applications

Business Case for Direct Peering  Typical Internet transit costs - $1000/Mbps per month  For 100 Mbps Internet transit then $100,000/mo  But coast to coast 100 Mbps channel is $1000/mo (e.g.  New optical technology will reduce that cost further  Compelling business case to do as much no-cost direct peering as possible  See  OBGP is a proposed protocol that will allow massive direct peerings  Each optical switch is in effect a mini-IX to allow direct no cost peering  OBGP will also automate peering relationships  Significant business opportunity for carriers who want to partner with CANARIE in building next generation optical Internet  For example Telia claims that they save 75% in Internet transit fees with massive direct peering  Speeds up convergence time on BGP routing

OBGP  Proposed new protocol to support control and management of wavelengths and optical switch ports  Control of optical routing and switches across an optical cloud is by the customer – not the carrier – true peer to peer optical networking  Use establishment of BGP neighbors or peers at network configuration stage for process to establish light path cross connects  Customers control of portions of OXC which becomes part of their AS  Optical cross connects look like BGP speaking peers – serves as a proxy for link connection, loopback address, etc  Traditional BGP gives no indication of route congestion or QoS, but with DWDM wave lengths edge router will have a simple QoS path of guaranteed bandwidth  Wavelengths will become new instrument for settlement and exchange eventually leading to futures market in wavelengths  May allow smaller ISPs and R&E networks to route around large ISPs that dominate the Internet by massive direct peerings with like minded networks

Opportunity for carrier and industry partners  To participate in a novel new Internet architecture that will allow customers to manage and control their own wavelengths anywhere across the network  Very attractive technology for Tier 2 ISP, research networks and ASPs  Yahoo and Cable and Wireless have already started down this path  It will allow them to create their own network topologies  To provide a valuable new service for customers that will allow them to reduce Internet transit costs by as much as 75%  To develop new value added services in IX brokering and management  To develop new fungible trading services in bandwidth trading and brokering  To experiment with new long haul optical technologies that will dramatically reduce cost of long haul transmission

CA*net 4 Deployment 1.CANARIE acts as a broker and acquires condominium wavelengths and condominium fiber from a number of carriers 2.CANARIE deploys optical switches across country interconnecting wavelengths (and a few routers at major peering points) 3.CANARIE assigns ownership and control of most of wavelengths and OXC ports to participating GigaPOPs 4.GigaPOPs can “optically peer” with each other at a given optical switch, using OBGP or other techniques 5.CANARIE works with GigaPOPs and participating carriers to develop Object Oriented Network techniques so that wavelengths can be traded in order to develop primitive wavelength exchanges 6.CANARIE works with GigaPOPs to provide “optical transit” service to each other by temporarily assigning wavelengths to another GigaPOP, using OBGP, OON or other techniques 7.CANARIE reserved wavelengths are used  To provide temporary optical transit service in support of Grid and eScience applications; and  To support a traditional layer 3 IP network interconnecting CANARIE routers

CA*net 4 Requirements  To partner with private sector to build national Internet optical distributed exchange points  See. LayerOne Distributed Optical Exchanges –  Condominium wavelengths and dark fiber where required to support new ultra long haul optics  Carrier neutral colo facilities for the interconnection and location of optical switch IXs  Additional wavelengths (or SONET channels) to interconnect layer 3 routers and aggregation services  Layer 3 service will be an optional service to GigaPOPs  Wavelengths across the Atlantic and Pacific  Native IPv6 network with IPv4 tunnels ( or dual stack)  Initially deploy off the shelf manually configured optical switches  Propose to deploy OBGP or similar technology to enable Object Oriented Networking and support massive BGP peerings as technology becomes available

CA*net 4 Possible Architecture Vancouver Calgary Regina Winnipeg Ottawa Montreal Toronto Halifax St. John’s Fredericton Charlottetown Chicago Seattle New York Europe Customer controlled optical switches Layer 3 aggregation service Optional Service Available to any GigaPOP Large channel WDM system

Wavelength Scenarios Vancouver Calgary Regina Winnipeg Toronto Halifax St. John’s Seattle Montreal Workstation to Workstation Wavelength University to University Wavelength CWDM BCnet RISQ GigaPOP to GigaPOP Wavelength Campus OBGP switch

Wavelength Setup AS 1 AS 2 AS 3 AS 4 AS 5 AS 6 Dark Fiber Wavelength Object owned by primary customer Wavelength Subcontracted by primary customer to a third party AS 1- AS 6 Peer AS 2- AS 5 Peer Regional Network University ISP router

Wavelength Logical Mapping AS 1 AS 2 AS 3 AS 4 AS 5 AS 6 Primary Route Backup Route AS 1- AS 6 Peer AS 2- AS 5 Peer Regional Network University ISP router

Resultant Network Topologies AS 1 AS 5 AS Regional Network University AS ISP router OBGP Potential OBGP Peering BGP Peering on switches at the edge Packet Forwarding in the core

CA*net 4 Architecture Details  Design philosophy is how optimize network design and architecture from the viewpoint of a BGP speaker  BGP is the fundamental peer to peer protocol between autonomous networks on the Internet  Each link on CA*net 4 is a BGP point to point peering connection between optical switches which act like Internet Exchange points  Each switch element is an independent virtual BGP router with its own router table, filtering, etc  The virtual BGP routers can be part of a private AS managed by CANARIE or they can be assigned to a customer and be a router within the customer’s AS  There are no spans, links or paths as in a traditional SONET or optical network  Link, path or span failure or outage is signaled by withdrawal or advertisement of BGP routes  Customer’s BGP route convergence determines restoral time  Customer responsible for maintaining route diversity  GbE is the defining reference protocol

Possible CA*net 4 Node 4 Channel GbE CWDM to local GigaPOP Carrier A Carrier B OC48 DWDM OC192 DWDM 8 Channel GbE CWDM to next CA*net 4 node 2xGbE 10xGbE Optional Aggregating Router Carrier A OBGP Switch CA*net 4 switch

Physical Wavelength Configs

Logical Wavelength Configs OBGP links ORAN B ORAN A ORAN C ORAN D Carrier A Carrier B GbE over CWDM CA*net 4 GbE over 10GbE over OC-192 DWDM GbE over 2GbE over OC-48 DWDM

Possible Wavelength Assignment  Illustrative purposes only  Assume 150 wavelength system across Canada  50 wavelengths assigned to provincial networks based on a number of criteria including ability to extend wavelengths into provincial network and requirements for high bandwidth applications  ORANs encouraged to extend wavelengths to individual institutions  Institutions encouraged to deploy optical switches  On all cross sections a minimum of 100 wavelengths dedicated to CA*net 4 and carrier partners  2 wavelengths dedicated to CA*net 4 layer 3 aggregation service (looks like old CA*net 3)  10 wavelengths (and OCX ports) reserved for temporary applications like Grids or eScience  A wavelength and OXC port bartering and exchange mechanism so that ORANs can swap wavelengths will be an important requirement

Example Physical Architecture  CANARIE builds heterogeneous network made from many sources e.g (illustrative purposes only):  dark fiber from St. John to Halifax using ULH 16 channel POS  dark fiber condo from Halifax to Fredericton using 16 channel 10GbE  Condo wavelengths from RISQ from Edmonston to Ottawa sharing 32 channel 10GbE system  dark fiber from Ottawa to Winnipeg with Onet using 16 channel POS at OC-192  Condo wavelengths from Bell Canada from Montreal to Chicago as part of a 140 wavelength system  wavelengths from Telus from Chicago to Winnipeg as part of a 140 wavelength system  dark fiber from GT Telecom from Winnipeg to Calgary using 16 channel 10GbE  wavelengths from Shaw from Calgary to Vancouver as part of a 32 channel 10GbE systems  wavelengths from 360 Networks from Halifax to London as part of a 400 wavelength system  Wavelengths from Teleglobe from Seattle to Honolulu – Sydney – Tokyo - Seoul

OBGP Variations 1.OBGP Cut Thru  OBGP router controls the switch ports in order to establishes an optical cut through path in response to an external request from another router or to carry out local optimization in order to move high traffic flows to the OXC 2.OBGP Optical Peering  External router controls one or more switch ports so that it can establish direct light path connections with other devices in support peering etc 3.OBGP Optical Transit or QoS  To support end to end setup and tear down of optical wavelengths in support of QoS applications or peer to peer network applications 4.OBGP Large Scale  To prototype the technology and management issues of scaling large Internet networks where the network cloud is broken into customer empowered BGP regions and treated as independent customers

OBGP Optical Peering  Primary intent is to automate BGP peering process and patch panel process  Operator initiates process by click and point to potential peer  Original St. Arnaud concept  Uses only option field in OPEN messages  Requires initial BGP OPEN message for discovery of OBGP neighbors  Virtual BGP routers are established for every OXC and new peering relationships are established with new BGP OPEN message  Full routing tables are not required for each virtual router  No changes to UPDATE messages  No optical transit as all wavelengths are owned by peer  Uses ARP proxy for routers on different subnets  Wade Hong Objects concept  Uses an external box (or process) to setup optical cross connects  SSH is used to query source router of AS path to destination router  Each optical cross connect is treated as an object with names given by AS path  Recursive queries are made to objects to discover optical path, reserve and setup  NEXT_HOP at source router is modified through SSH  End result is a direct peer and intermediate ASs disappear  Requires all devices to be on same subnet

OBGP Optical Transit  Wavelengths are under control of another entity who has temporarily allowed them to be available for transit  Viagenie – Marc Blanchet and Florent Parent  Designed specifically for optical transit applications  Uses MBGP and establishes new address family for OBGP  Community tags are used to advertise availability of optical paths as part of NLRI and COMMUNITY TAG  Reservation and setup is done by advertising update NLRI message  Exploring using CR-LDP & RSVP-TE with AS loose routing for path reservation and setup  Changcheng Huang  The same NLRI message is sent back and forth and modified to indicate first availability of wavelengths, reservation and setup  Over rides loop back detection in RIBS for advertised NLRI messages

Target Market for OBGP  University research and community networks who are deploying condominium fiber networks who want to exchange traffic between members of the community but who want to maintain customer control of the network at the edge and avoid recreating the need for aggregating traffic via traditional mechanisms  E.g. Ottawa fiber build, Peel County, I-wire, SURAnet, G-Wire, CENIC DCP, SURFnet, etc etc  Next generation fiber companies who are building condominium fiber networks for communities and school boards and who want to offer value added fiber services but not traditional telcommunications service  E.g. C2C, Universe2u, PF.net, Williams, QuebecTel, Videotron, etc  Next generation collocation facilities to offer no-cost peering and wavelength routing  Metromedia, Equinix, LINX, PF.net, LayerOne, Westin, PAIX, Above.com, Colo.com, etc etc  Over 500 Ixs and carrier hotels worldwide

OBGP Peering  Possible technique for allowing automatic peering at IXs between consenting ISPs  External routers are given control of specific ports on the OXC  The router that controls switch can act as an optical route server notifying all peers of any new consenting OBGP peers  External routers signal to each other if they wish to setup direct optical connection  Choice of partner can be based on size of traffic flows  Partners can be changed through a routing flap  Only see each other’s customers routes – not the default core

OIX using OBGP AS AS AS AS Institution A Institution B Institution C Institution D Figure 10.0 Switch Ports are part of institution’s AS

Transport Architecture  Heterogeneous transport architectures used on backbone links  Type of transport architecture on each link determined by length of link between O-ADMs, GbE-ADMs or OBGP switches, requirement for optical repeaters or regenerators, etc  Examples:  8 or 16 channel GbE used on short haul links (up to 2000 km) between OADMs or OBGPS; or  OC-192 Ethernet over SONET with multiplexed 10 single GbE or trunked 10GbE; or  Proprietary 8 channel 2 x GbE multiplexed into OC48 optics with FEC wrapper  Repeaters:  GbE or 10GbE 2R transceivers every km combined with GbE or 10Gbe 3R switches every 200 – 400 km; or  Traditional EDFAs at 1550nm every km with OC-192 regenerators at every km; or  All optical broadband: Counter rotating Raman amplifiers, multi band EDFAs, EFFs, dispersion correction fiber, etc

Tributary Architecture  Customer can connect through OADM,Gbe-ADM, direct to OBGP switch or through CA*net 4 router  Customer access link is either GbE or trunked 10GbE (I.e. 10 separate GbE channels  In future customer will have a choice of protocols, but for now GbE will be basic standard across the network

Switch Architecture  Low speed MEMs or similar capacity switch  Could also use non blocking GbE switch  Switch can also be distributed across an optical network using GMPLS or ODSI  Each switch component can be controlled by a socket/port by any external network element with appropriate security mechanisms  If OXC used for traffic engineering or QoS then controlling router manipulates both input and output ports  If OXC used for distributed peering then participating AS only owns either INPUT or OUTPUT ports  Eventually switches can also support optical trunking of many optical paths  Switch commands are kept very simple, leaving all complexity to OBGP messages  Switch does not know or care the direction of the wavelength – that is established with OBGP protocol