September 1998 http://www.canet2.net CANARIE Inc “Canada’s National Optical Internet” September 1998 http://www.canet2.net turcotte@canarie.ca http://www.canarie.ca.

September 1998 http://www.canet2.net
CANARIE Inc “Canada’s National Optical Internet” September 1998 Tel:

CA*net 3 World’s first national optical Internet
First Internet network built from the ground up to support Internet first, voice second All existing Internet networks are built on technology originally designed for voice - e.g. SDH/SONET & ATM Consortium members include Nortel, Newbridge, Cambrian, CISCO, Bell, etc Key features: use of individual DWDM wavelengths directly coupled to routers Use intrinsic self healing capabilities of Internet and eliminate SDH/SONET and ATM layers MPLS for layer 3 restoral, protection and traffic engineering

National Optical Network
CA*net 3 GigaPOP RAN WURCnet SRnet MRnet OC3 DS3 OC12 BCnet ACORN St. John’s OC3 Calgary Regina Winnipeg RISQ Charlottetown ONet OC48 Fredericton OC12 Teleglobe Montreal Halifax Vancouver Ottawa STAR TAP Toronto Chicago 10 10 9 10 9 9 10 10 10 10 10 10 10

What is an Optical Internet?
WDM fibers where individual wavelengths are the link layer interconnect directly connected to routers via Optical ADM (Add Drop Mux) or WDM coupler High Performance Router acts as the main switching routing device Bypass or cut-thru connections via dedicated wavelengths SONET or Gigabit Ethernet framing (also 10xGbE) Use intrinsic self healing nature of Internet for redundancy and protection (don’t require SONET/SDH layer) Traffic engineering and network management done via MPLS Network design optimized for unique characteristics of Internet traffic

Why build an Optical Internet?
Dramatic growth in IP traffic ISPs are already starting to deploy OC-48 IP networks Customers are starting to order OC-12 IP local loops How soon before we need OC-192 or OC-768 IP?? Future trends indicate IP growth will continue IP telephony could be very, very big New Internet 2 and CA*net 2 applications Internet characteristics significantly different than traditional telecommunications traffic If IP is the dominant traffic then optimize network design for IP CA*net 3 will be world’s first network designed from the ground up to carry first and foremost, Internet traffic

Traffic Growth Data Voice Data is 23x Voice Traffic Data is 5x
Source:Lightwave April 1998

Scary Statistics UUnet Growth Rates: deployed first OC3 in early 1996
deployed OC-12 early 1997 will deploy OC-48 late 1998 will deploy NxOC-48 late 1999 engineering has asked vendors for 1000 x OC-12 (Terabit) for next year UUnet deploys a new T3 every day MCI has started connecting up first OC-12 customers Qwest has started offering OC-48 interconnect to the Internet

Scary Statistics Number of customers in Canada want T3 Internet
Data network volume will exceed voice in By voice will be 1% of all traffic Christin Huitema - Bellcore Internet Traffic is growing around 10% a month this means we need 35 Tbps in US by 2001 this means we need 3 Tbps in Canada by 2001 Number of major carriers will offer POS in MCI, Sprint, Teleglobe, Bell Level 3, Qwest, etc We have a history of underestimating Internet growth

Where will the traffic come from?
Maybe new splitterless ADSL- UDSL if UDSL reaches same penetration as modems then network capacity will have to increase by 100 to 1000 times The future of the Internet is not multimedia - Mike O’Dell Computer appliances talking to each other FedX is already a Terabit network thousands of disks and tapes shipped daily jitter and delay is pretty poor cost for shipping tape approx cents/byte Current cost of sending data over fiber .001cents/byte By 2001 telecommunication cost will be close to FedX cost

The real driver for Optical Internet
Traditional OC-48 SDH/SONET network costs about $US $5000 km per year before overhead, engineering and maintenance 20 year amortization on fiber and installation 5 year amortization on optical amps, regen, SONET Mux, etc Optical Internet with today’s technology costs about $US 500-$750 per kilometer per year With low cost regen (e.g.10xGbE), low dispersion fiber, and long range optical amplifiers optical Internet will cost $US $200 per km per year Optical Internet also has significantly less overhead, engineering and maintenance costs. see Engineering paper for financial analysis

Opportunity for Canada
World leader in SONET/optical networking - JDS Fitel, Nortel, Cambrian, Positron Fiber Systems, CISCO Canada, PMC Sierra, QNX Over 75% of the world’s Internet traffic is carried on equipment made in Canada CISCO GSR12000 SONET I/F made in Ontario -95% market Nortel Optical Transport made in Montreal - 75% market Newbridge ATM switches made in Ottawa - 50% market JDS Fitel optical components made in Ottawa -85% market Possibility of leveraging our technology and leadership to increase export opportunities and job growth in this area A network for basic research unparalleled anywhere in the world

Definition of bandwidth
We will use data definition instead of telco definition Data definition assumes half duplex broadcast media e.g. Ethernet means the aggregate capacity for both sending and receiving data Telco definition assumes full duplex non broadcast media plus reserved protection bandwidth means the capacity in one direction only on the working path OC-48 (2 Gbps) unprotected telco = OC-96 (4 Gbps) data OC-48 UPSR telco = OC-192 (16 Gbps) data OC-48 4/BLSR telco = OC-384 (32 Gbps) data

CA*net 3 Objectives CANARIE is an “industry led” consortium
Internet 2 is a “university led” consortium NGI is US government initiative to support pre-competitive R&D Roughly same objectives with slightly different emphasis on role of industry participation CANARIE’s objectives for C3 similar to C2 To facilitate a partnership between industry, carriers, RAN’s and R&E community in order to accelerate the deployment of next generation commercial Internet products and services; and To catalyze the building of a sustainable virtual high performance R&E network

Design Principles Must meet needs of carrier for eventual commercial deployment Showcase different Canadian technologies from CISCO, Newbridge, Nortel, etc R&E will eventually be one of many “communities of interest” on the network As per Hickling survey most researchers want a “production” network as opposed to a “test” network so new network technologies that may affect production like IPv6 will have to be done on a separate test network some network technologies (MPLS and QoS?) maybe can be tested in a production environment As per Internet 2 - “Production before experimentation”

Acceptable Use Policy Same AUP as CA*net 2
Any Canadian organization that is doing high performance meritorious research or applications development that cannot be carried out on the commercial Internet CA*net 3 will only interconnect GigaPOPs One GigaPOP per province plus Ottawa - others may be added GigaPOPs interconnect to regional high speed networks Same Tier A/B/C policy as C2 Allows CA*net 3 to peer with similar international research networks like Abilene, vBNS, etc All institutions must maintain separate commercial Internet connection

RFI Criteria CANARIE RFI process was to select a “research partner” to deploy a national optical Internet it was NOT a “fee for service” network RFI Responses were evaluated on: strength of proposed R&D program participation of Canadian industry partners ability to deploy an optical Internet demonstrated ability to partner with R&D community respondent’s own contribution to the project cost to CANARIE Four respondents to core backbone RFI As a costing reference respondents were asked for a 2 wavelength (Tx/Rx) hop by hop network, coast to coast

Winning Proposal Bell Canada consortium
CANARIE pays 1/3 of WDM costs & 100% of router, regen and ATM costs no charge for OA&M costs partners include Nortel, CISCO, JDS, Cambrian and Newbridge the carrier is building a parallel commercial POS service on same fiber so can spare much of the same equipment up to 8 OC-192 available to CANARIE e.g. 80 Gbps “capable” network Budgetary WDM Costs for 2 wavelength Vancouver to Montreal $7.8m for OC-48 (Regen $100k) $13.5m for OC-192 (Regen $250k) $7.6m for OC-48 from Montreal to Halifax

Dramatic Drop in Costs Assume 48 strand fiber, but only 24 strands used and 8 wavelengths per fiber and 10% cost of money Fiber cable and installation costs: Per cable Per Fiber: $6 per 20 year amortization $ Installation: $15 per 20 year amortization Repeater huts: 20 year amortization Optical amplifier and WDM couplers are required for each separate fiber strand Optical Amplifier: 5 year amortization WDM couplers: 5 year amortization For each strand of fiber we will assume that an 8 wavelength WDM system Regen: 5 year amortization / Total per wavelength cost $ .35 per meter ($.17 x 2) CA*net 3 approximately 5700 km x 2 wavelengths cost is about $4 million per year With lower cost regen and new optical amps cost could drop to less than $1m per year Traditional ATM over SONET network would cost $37m per year

National IP/WDM Network
Edmonton Saskatoon Additional OC-192 WDM Routes for future use 4/BLSR Winnipeg Ottawa Calgary Regina Montreal Charlettown 4/BLSR Vancouver St. John’s 4/BLSR Fredericton Teleglobe Toronto - CANARIE Drop Site Chicago Halifax 8 Wavelengths per route 4 reserved for traditional SONET 4/BLSR by carrier CANARIE OC-192 Route CANARIE OC-48 Route

Recommendation Initial OC-48 WDM 2 wavelength network hop by hop router network using SONET framing on Nortel WDM gear between Vancouver and Halifax with no alternate routing paths explore mutual exchange of alternate backup paths & routing with vBNS and Abilene GigaPOPs are initially responsible for alternate routes in the event of a network failure. R&D for fast routing recovery will have to be done at GigaPOPs CANARIE will work with GigaPOPs to explore how regional networks could be used as an alternate path for C3 CANARIE provides provincial allocation for local loops CANARIE pick up ATM costs for Teleglobe, Charlottetown, Chicago and St. John’s and for transition CA*net 2 connectivity until local WDM node is operational CANARIE reserves funds for Gigabit Ethernet alternate paths and general applications R&D

Issues Carrier will be deploying parallel POS network so no need to commercialize before 2002 BUT early commercialization may allow for extended use of “shared” network beyond 2002 addition of redundant routes can be be cost shared No redundancy or alternate paths - so subject to single point failure outages from routers and fiber cuts CANARIE maintain CA*net 2 until 2002, or; GigaPOPs responsible for alternate path, or; CANARIE deploy alternate national paths, or; Explore exchange of alternate paths with Abilene and vBNS e.g. drop OC-12 POS to Seattle & NY

Possible Alternate Routing Paths
The Power of Hybrid Architecture Future Combined Alternate Path for C3 and Regional High Speed Network Edmonton Saskatoon Future Combined Alternate Path for C3 and Provincial High Speed Network Winnipeg Future CSI Alternate Path via CA*net 2 Ottawa Vancouver Calgary Regina Charlettown St. John’s Seattle Montreal Toronto Fredericton Teleglobe Chicago Halifax New York Abilene or vBNS Alternate Path Backbone routers

Alternate Path Routing Issues
Regional Network becomes a transit network rather than a leaf network significant traffic engineering, management and policy issues over control of routers, loading, etc CANARIE funds can then be used to support regional backbones Examples of alternate path and funding BCnet and CANARIE build shared pipe to Seattle RISQ and CANARIE build shared pipe to NY ONet and CANARIE build shared pipe across Ontario OCRInet, Teleglobe and Dalhousie build CSI path between Ottawa and Nova Scotia MPLS will be an absolute necessity for traffic engineering Can be done a lot easier with ATM

Network Bandwidth vs Moore’s Law
Bandwidth Doubling every 6 months CPU Power Doubling every 18 months

Fractal Internet OC3c OC3c 1 user Average Load Average Load 100 users
Need big buffers or big bandwidth Average Load Average Load 1 million users Traditional Voice Traffic Internet Traffic

Tx/Rx Asymmetry 20:1 Cnet Regional Network 4:1 To Other Regionals 6:1
Big Server e.g. Microsoft e.g. Netscape Backbone Network Regional Network Cnet To Other Regionals 20:1 3:1 4:1 6:1 2:1 Tx:Rx

Web Points of Congestion
DNS 13% Connect 12% Network 42% Server 33% Bellcore Study C Huitema 23/1/97

Speed of Light is a major limitation Bandwidth-Delay Product Issues
On long haul optical networks speed of light becomes limiting factor with a normal TCP connection from Vancouver to Toronto maximum throughput is about 7 Mbps The bigger the pipe, the slower the throughput!! Multimedia “high twitch” services require an ACK-NAK protocol for reliable transfer & event synchronization e.g. TCP, SNA IP uses TCP for connection management on big pipes need to customize window size and MTU size RFC 1323 TCP-LW and RFC 2018 SACK Fundamental to advanced networking - still remains an unresolved problem with X-servers

Three types of traffic Human to Human Human to Computer
real time voice and video, tele-medicine, tele-immersive VR, etc very sensitive to jitter and delay very symmetric & growing linearly Human to Computer web, voice mail, video servers, call centers, fax - mostly TCP jitter and delay can be compensated with client buffering fractal & very asymmetric & growing exponentially Computer to Computer bio-informatics, IP appliances, off site backups, web caching insensitive to jitter and delay, but extremely fractal extremely asymmetric & growing exponentially plus

Implications Future traffic could be high volume, high fractal TCP “computer to computer” with lots of empty space for other types of traffic Large peak to average loads to accommodate fractal nature of Internet Smaller volume, jitter sensitive “human to human” traffic can be inserted in empty space prioritized with simple QoS mechanisms Network reliability and performance must be defined from a systems level, not a network level Throughput and congestion are increasingly server bound, not network bound So high bandwidth IP pipes with simple QoS and reliability mechanisms may be all that is needed

Possible CA*net 3 Architecture
GigaPOPs to minimize hot potato routing and shortest path to destination also provide redundancy, load sharing, reliability Asymmetric WDM to compensate for asymmetric Tx/Rx Use both sides of fiber ring As long as high priority traffic less than 50% of aggregate bandwidth Because Internet is server bound and fractal use tools like WRED to flow control TCP traffic MPLS for explicit routing at GigaPOPs Communities of Interest (COI) using MPLS to provide VPN like service Distributed hierarchical caching service integrated with routers to relieve server congestion Coarse grained QoS using Differentiated Services is “good enough”

Optical Internet Architecture
Both sides of 4/BLSR 1:1 span ring used for IP traffic Traditional SONET Mux or DCS Traditional SONET Mux or DCS WDM WDM 3 0C-48 Tx 2 OC-48 Rx High Priority Traffic Cannot exceed 50% of bandwidth in case of fiber cut Asymmetric Tx/Rx lambdas that can be dynamically altered Traditional SONET Restoral Low priority traffic that can be buffered or have packet loss in case of fiber cut

Hybrid Optical Internet Using ATM as a Return Path
Working side of 4/BLSR 1:1 span ring used for IP traffic OADM SONET Traditional SONET Restoral Traditional SONET Gear ATM network used as a return path to optical IP forward path Asymmetric Wavelengths Large Traffic Flows in this direction

Hybrid Optical Internet Using ATM as a Restoral Path
Working side of 4/BLSR 1:1 span ring used for IP traffic OADM SONET Traditional SONET Restoral Traditional SONET Gear ATM network used as a backup path to primary IP path in case of network failure

Example Hybrid Network
US Gateway Toronto Montreal Calgary Vancouver ATM Cloud for regular IP and QoS WDM used to carry bulk high volume traffic in one direction

Optical Circuits or Datagrams?
Photonic packet switching (datagrams) still a long way off requires extreme high speed optical switching Optical WDM circuit switching makes sense: For crafts maintenance & servicing Automatic provisioning of physical links makes sense in municipal networks still too costly for long haul Optical Circuit Data Switching (like ATM) circuit setup time X bandwidth larger than size of data flow

Layer 3 Restoral IP network is intrinsically self healing via routing protocols By cranking down timers on interface cards and keep alive message time-out we can achieve same restoral speed as SONET Biggest delay is re-calculation and announcement of changes in routing tables across the network MPLS promises to simply the problem maintain a set of attributes for restoral and optimization may provide a consistent management interface over all transport services -WDM, SONET/SDH, ATM, Frame Relay, etc Layer 3 restoral allows for more intelligent restoral can use a hybrid mix of restoral and protection circuits Can use QoS to prioritize customers and services Only UDP packets (e.g telephony) require fast restoral allows simultaneous use of both working and protection circuits

Example Backbone Architecture
Protection Fiber Working Fiber US Gateway Toronto Montreal Calgary Vancouver By pass WDM Traditional Hop by Hop Routing

Carrier Node Carrier Tributary SONET services - OC3c, OC12c, etc WDM
Coupler WDM Coupler DCS TransportNode Traditional SONET OC-48/192 Working Fiber Working Fiber To Local GigaPOP (ATM, SONET, WDM etc) To Local GigaPOP (ATM, SONET, WDM, (etc) Carrier Router Transponders WDM Coupler WDM Coupler Electrical Regenerator OC-48/192 Protection Fiber Cut thru asymmetric Lambdas to next Router Protection Fiber

Regional Optical Network
Central Office To Commercial Internet To CA*net 3 Packet over SONET GigaBit Ethernet Dual Redundant Paths - can be switch protected or dual path OADM OXC? Local WDM Fiber Ring Provided by Cable Company or Telco University A Router Ethernet ATM OADM OADM Reuse of same wavelength University B ATM Analog Video OADM GigaPOP Router

Example Physical Layer
CA*net 2 Research Institute A Telco Research Institute B DWDM CA*net 3 ATM CSI Route Policy Server ATM Wireless RAN ATM DWDM University B Community College University A Wireless Distributed Municipal GigaPOP

Example IP Layer AS ##1 AS ##2 CA*net 2 Regional Institutional GigaPOP
Telco CA*net 3 PNNI X.x.x.x/ X.x.x.x/ iBGP CSI Route Policy Server iBGP RAN eBGP X.x.x.x/ OSPF Wireless OSPF X.x.x.x/ OSPF University B X.x.x.x/ Intermediate Cache X.x.x.x/ Community College University A Wireless High School or CAP site AS ##3 BGP Confederation ### Distributed GigaPOP Daughter Cache x.x.x.x/

Optical Internet Exchange
Common Internet Exchange Router Packet over SONET ISP A Ethernet ATM OADM ISP B Web Server ISP C Small ISPs

Optical Internet Exchange Logical Diagram
ISP C Common Internet Exchange Router ISP A ISP B Web Farm Small ISPs

Possible CSI implementation
CA*net 3 path for best efforts Internet Internet CSI Core Forwarder 36190 CSI Core Forwarder 36190 CA*net 2 ATM for QoS & VPN Central Office Central Office GigaPOP B CSI Route Server Regional ATM Network Regional ATM Network GigaPOP A CSI Route Server Ethernet ATM University A Router CSI Edge Forwarder University B Router CSI Edge Forwarder

Campus Optical Network
Analog Video Building C ATM OXC NxN Building A ATM Building B Ethernet OADM Re-use same wavelength OADM G ATM Gigabit Ethernet OXC NxN 32 wavelengths x Number of fibers Each wavelength can have a different service with complete data transparency and be independently managed. Video can be switched between many sources and destinations with OXC Campus Border Router To GigaPOP

Major Long Haul Costs $200k per Tx/Rx $200k per Tx/Rx WDM WDM Coupler
50 km Wideband Optical Repeater $250K SONET Regenerator or Gigabit Ethernet regen SONET Regenerator or Gigabit Ethernet regen 250 km Approximate Distances for OC-192 system

Regional Network Costs
Fibre - $4 to $6 per meter 20 year economic life and 10% maintenance per year 48 strands NZDSF Right of way $0 -$10 per meter per year common solution is to offer right way owner free use of a couple of strands instead of paying cash Installation $25 per meter underground in cities $6 per meter on poles (maintenance costs 20% year) Twenty year amortized cost with 10% cost of money plus maintenance plus right of way costs at $10/year per meter $17 per year per meter for 48 strands $1.50 per year per meter per strand $.50 per year per meter per wavelength A 5 km OC-48 local loop should cost about $2500/year

Opaque or Transparent Networks?
Long haul networks will have to electrically and optically opaque EDFAs are very sensitive to changes in optical power levels on different wavelengths Switched wavelengths or wavelength translation can affect power levels on other wavelengths Electrical regen equipment needs a protocol for clocking and timing e.g. SONET or Gigabit Ethernet Municipal Networks without EDFAs can be optically transparent and therefore optical connection to customer premises possible Municipal Networks with EDFA and without electrical regen equipment can be electrically transparent and carry any type of electrical protocol including analog video but a electrical interface required at customer premises to protect power levels on EDFAs

Gigabit Ethernet Framing
Gigabit Ethernet Framing advantages frame size = packet size therefore packet switching and SAR more efficient and easier to implement data format consistent with LAN format with no translation low cost tributary service - do not need to terminate link on a router or SONET DCS equipment new 10xGigabit Ethernet will equal OC-192 standard SNMP MIBs, but not accessible by out of band interoperable standard from many vendors No scrambling sync or packet loss Gigabit Ethernet Framing disadvantages not very efficient with 8B/10B block coding new 10xGigabit Ethernet may use more efficient coding No standard out of band management or monitoring But some WDM suppliers provide this

SONET Framing SONET framing advantages SONET framing disadvantages
well established jitter specifications out of band management systems can be used in SONET networks for fast restoral and protection very high efficiency - over 98% SONET framing disadvantages no interoperable standard SAR processing more complex as there can be multiple packets per frame, or packets can cross frame boundaries tributary services require SONET mux services no well established carrier network management protocols for fault detection and location, especially on long haul when SONET used in independent links

Gigabit Ethernet vs SONET
Switch SONET Router Tributray With SONET links everything must terminate on SONET gear - either routers or DCS With Gigabit Ethernet costs are much less and you can backhual bridged devices to and expensive high performance router $50K G $25K Gigabit Ethernet Regen Tributray with Bridged Switch $125k $125K $250K

Third Generation Router
Terabit routing - 32 x OC “Tiny Tera” Juniper, Pluris, Avici, GSR 12000 5 Tiny Tera = all existing switch capacity in North America Routing on a chip with no caching wire speed routing with HPCC techniques ATM in the backplane, packet in the wire BGP+, CBQ and MPLS Only “let it smoke” packets Integrated OADM with SONET path/link protection services and Fast IP framing Advanced routing functions like multicast, interior routing be handled by GigaPOP or CPE router

Future Optical Internet Integrated Transport Services
Different Protocol Stacks Integrated to provide different size bandwidth pipes and CoS HDWDM OC-3084 OXC ADM ATM/IP Network IP SONET Network IP Optical Network OADM IP/ATM Network IP over ATM IP Optical IP Sonet QoS & VPNs up to OC3 OC3, OC12, OC48 Greater than OC-48 60 11

Future Optical Internet MPLS as common management layer
OC3, OC12 IP over ATM IP Optical IP Sonet QoS & VPNs up to OC3 OC-48, OC-192 DWDM LSP ATM VCs SONET LSP 60 11

Communities of Interest
Community of Interest is a “layer 4” VPN separate BGP routing tables for each community explicit routing and switching can provide higher bandwidth and QoS for a given community of interest community may include value added services such as e-commerce different than VPN as “community” is not necessarily isolated from Internet Internet is just another community Future IP networks may support many communities of interest K-12, Health, Bio-informatics, High Energy Physics, Post production, Automobile Engineering, etc individuals or institutions may be members of multiple different communities Communities can easily extend across competitive networks

Communities of Interest
Science Research Community K-12 Community ISP C ISP B Carrier A Health Community GigaPOP Carrier B ISP A ISP D Banking Community 60

R&D Issues Routers c/w OC-48 Interfaces into OC-192 transponders?
Routing Convergence on Fiber Cut? (OSPF, IS-IS)? UDLR asymmetric routing? Routers in CO or use local optical loop with OXC to locate routers in GigaPOP? Hybrid Optical c/w SONET, ATM? Data striping on WDM? Wavelength to wavelength routing? Cut through WDM or SONET, or label switched WDM or SONET OTDM - Framing protocols other than SONET? Flow control in the event of a fiber cut - WRED? Tributary services into RAN - ATM, SONET, WDM, Gigabit Ethernet?

September 1998 http://www.canet2.net CANARIE Inc “Canada’s National Optical Internet” September 1998 http://www.canet2.net turcotte@canarie.ca http://www.canarie.ca.

Similar presentations

Presentation on theme: "September 1998 http://www.canet2.net CANARIE Inc “Canada’s National Optical Internet” September 1998 http://www.canet2.net turcotte@canarie.ca http://www.canarie.ca."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

September 1998 http://www.canet2.net CANARIE Inc “Canada’s National Optical Internet” September 1998 http://www.canet2.net turcotte@canarie.ca http://www.canarie.ca.

Similar presentations

Presentation on theme: "September 1998 http://www.canet2.net CANARIE Inc “Canada’s National Optical Internet” September 1998 http://www.canet2.net turcotte@canarie.ca http://www.canarie.ca."— Presentation transcript:

Similar presentations

About project

Feedback