Outline A brief Historical aside Review of Transmission (Transport) Technologies, Architectures and Evolution Transporting Broadband across Transmission Networks designed for Narrowband Current Issues: Broadband IP Transport Analysis Ongoing Investigations in IP/OTN Networks

A Brief Historical Aside

LDBell-LabsWEBCSME AT&T LDAT&T Labs AT&T circa 1997 Bell-LabsWEBCSME Lucent circa 1997 Pre 1984 AT&TBOCsLDBell-LabsWEBCSME AgereAvaya RBOCs circa 1984 US West Ameritech SouthWest Bell Bell South Nynex Bell-Atlantic Pac Bell Bellcore Qwest Telcordia Tellium SBC Verizon Bell South AT&TLucent The Bell System Legacy Today

Review of Transmission (Transport) Technologies, Architectures and Evolution

Opening Trivia Question What is the difference between a DS3 (or DS1) and a T3 (or T1)?

Asynchronous Data Rates Digital Signal Level 0 DS0 64 Kb/s internal to equipment Digital Signal Level 1 DS Mb/s intra office only (600 ft limit) Digital Signal Level 3 DS3 45 Mb/s intra office only (600 ft limit) T1 Electrical (Copper) Version of DS Mb/s repeatered version of DS1 sent out of Central Office T3 Electrical (Copper) Version of DS3 45 Mb/s repeatered version of DS3 sent out of Central Office

Asynchronous Digital Hierarchy DS1 DS3 Asynchronous Optical Line Signal N x DS3s 28 DS1s = 1 DS324 DS0s = 1 DS1 DS0 (a digitized analog POTS 64 Kbits/s) Asynchronous Lightwave Systems typically transport traffic in multiples of DS3s i.e.... 1, 3, 12, 24, 36, 72 DS3s DS0

Asynchronous Networking Manual DS1 Grooming/Add/Drop LW M13 DSX3DSX3 DS1 M13 DSX1DSX1 DSX1DSX1 DSX3DSX3 LW Manually Hardwired Central Office No Automation of Operations Labor Intensive High Operations Cost Longer Time To Service DS3

Some Review Questions What does the acronym SONET mean? What differentiates SONET from Asynchronous technology? What does the acronym SDH mean?

The Original Goals of SONET/SDH Standardization Vendor Independence & Interoperability Elimination of All Manual Operations Activities Reduction of Cost of Operations Protection from Cable Cuts and Node Failures Faster, More Reliable, Less Expensive Service to the Customer

SONET Rates DS3s are STS-1 Mapped DS3 STS Mbits/s SONET Optical Line Signal OC-N = N x STS-1s N is the number of STS-1s (or DS3s) transported 28 DS1s = 1 DS3 = 1 STS-124 DS0s = 1 DS1 (= 1 VT1.5) DS1 DS0 (a digitized analog POTS 64 Kbits/s) DS0

OC level STM level Line rate (MB/s) OC OC-3 STM OC-12 STM OC-48 STM OC-192 STM SONET and SDH

STE LTE PTE PTE PTE STE LTE PTE PTE PTE DS-3 DS-3 DS-3 DS-3 DS-3 DS-3 OC-3 TM SONET Line SONET Path SONET Section TM = Terminal Multiplexor DS = Digital Signal PTE = Path Terminating Element LTE = Line Terminating Element STE = Section Terminating Element SONET Layering for Cost Effective Operations

SONET Point-to-Point Network Repeater TM Section Line Path STS-1 Frame Format Line Overhead Section Overhead Path Overhead STS-1 Synchronous Payload Envelope STS-1 SPE

SONET Ring Network Architectures

Unidirectional Path Switched Ring A-B A-B B-A B-A Path Selection W P fiber 1 fiber 2 A B C D Failure-free State Bridge Bridge

Bidirectional Line Switched Ring AÔCAÔCAÔCAÔC C ÔA AÔCAÔCAÔCAÔC Working Protection 2-Fiber BLSR B A D C

Some Review Questions Which SONET Ring Network is simpler? Which SONET Ring Network is inefficient for distributed demand sets?

Typical Deployment of UPSR and BLSR in RBOC Network Regional Ring (BLSR) Intra-Regional Ring (BLSR) Access Rings (UPSR) WB DACs BB DACs WB DACS = Wideband DACS - DS1 Grooming BB DACS = Broadband DACS - DS3/STS-1 Grooming Optical Cross Connect = OXC = STS-48 Grooming DACS=DCS=DXC

Emergence of DWDM Some Review Questions What does the acronym DWDM mean? What was the fundamental technology that enabled the DWDM network deployments?

First Driver for DWDM Long Distance Networks WDM NE Limited Rights of Way Multiple BLSR Rings Homing to a few Rights of Way Fiber Exhaustion BLSR Fiber Pairs

Key Development for DWDM Optical Fiber Amplifier 120 km OC-48 OLS TERM OLS RPTR OLS RPTR OLS TERM 120 km Fiber Amplifier Based Optical Transport - 20 Gb/s OC-48 Conventional Optical Transport - 20 Gb/s 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR TERM 40km 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR TERM 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR TERM 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR TERM 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR TERM 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR TERM 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR TERM 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR 1310 RPTR TERM OC-48 Increased Fiber Network Capacity

Transporting Broadband across Transmission Networks designed for Narrowband

T1/T3/OC3 FRS and CRS ATM Access ATM Access ATM Switch Public/Private Internet Peering ATM Access ATM Access Router T1/T3 IP Leased-Line Connections Core Router Core Router Access Router Access Router ATM Access RAS Access Router Access Router EtherSwitch RAS Core Router Core Router Backbone SONET/WDM RAS Farms T1/T3 FR and ATM IP Leased-Line Connections ATM Switch ATM Switch ATM Switch ATM Switch Core Router Core Router Data SP

High Capacity Path Networking Existing SONET/SDH networks are a BOTTLENECK for Broadband Transport Most Access Rings are OC-3 and OC-12 UPSRs while most Backbone Rings are OC-48. Transport of rates higher than OC-48 using the existing SONET/SDH network will require significant and costly changes. Clearly upgrading the SONET/SDH network everytime broadband data interfaces are upgraded based increased IP traffic is not an appropriate solution. Existing SONET/SDH networks are a BOTTLENECK for Broadband Transport Most Access Rings are OC-3 and OC-12 UPSRs while most Backbone Rings are OC-48. Transport of rates higher than OC-48 using the existing SONET/SDH network will require significant and costly changes. Clearly upgrading the SONET/SDH network everytime broadband data interfaces are upgraded based increased IP traffic is not an appropriate solution. Existing SDH-SONET Network IP router STS-3c STS-12c/48c/...

IP/SONET/WDM Network Architecture Core IP Node EMS... SONET ADM/LT OC-3/12 [STS-3c/12c] OC-12/48 OC-3/12 [STS-3c/12c/48c] SONET Transport Network SONET NMS Core IP Node EMS... Access Routers/ Enterprise Servers OC-48 SONET ADM/LT SONET XC WDM LT WDM LT 1, 2,... OC-3/12/48 [STS-3c/12c/48c] Pt-to-Pt WDM Transport Network OC-3/12/48 [STS-3c/12c/48c] OTN NMS IP = Internet Protocol OTN = Optical Transport Network ADM = Add Drop Multiplexor WDM = Wavelength Division Multiplexing LT = Line Terminal EMS = Element Management System NMS = Network Management System

Optical Network Evolution mirrors SONET Network Evolution Multipoint Network WDM Add/Drop Point-to-Point WDM Line System Optical Cross-Connect WDM Networking OXC i WDM ADM k      

IP/OTN Architecture Core Data Node EMS... OXC mc: multi-channel interface (e.g., multi-channel OC-12/OC-48) mc Optical Transport Network OTN NMS Core Data Node EMS... Access Routers Enterprise Servers OXC Core Data Node EMS... mc IP = Internet Protocol OTN = Optical Transport Network OXC = Optical Cross Connect WDM = Wavelength Division Multiplexing EMS = Element Management System NMS = Network Management System

Broadband IP Transport Analysis Credits to Debanjan Saha and Subir Biswas

Architectural Alternatives IP-over-DWDM: IP routers connected directly over DWDM transport systems. IP-over-OTN: IP routers interconnected over a reconfigurable optical transport network (OTN) consisting of optical cross-connects (OXCs) connected via DWDM.

Architectural Alternatives

Quadruple Redundant Configuration of IP Routers at PoPs Currently deployed by carriers to increase router reliability and perform load balancing. Upper two routers are service routers adding/dropping traffic from the network side and passing through transit traffic. Lower two routers are drop routers connected to client devices. Two connections from the network port at the ingress upper (service) router to two drop ports, one in each of the lower (drop) routers. Client device sends 50% of the traffic on one of these drop interfaces and 50% on the other (it is attached to both of the drop routers). Not required for OXCs.

IP-over-DWDM: Pros and Cons IP-routers with OC-48c/OC-192c interfaces and aggregate throughput reaching 100s of Gbps. Transport functions like switching, configuration, and restoration are moved to the IP layer and accomplished by protocols like MPLS, thus providing a unifying framework. IP routers control end-to-end path selection using traffic engineering extended routing and signaling IP protocols. Supports the peer-to-peer model where IP routers interact as peers to exchange routing information. Can router technology scale to port counts consistent with multi-terabit capacities without compromising performance, reliability, restoration speed, and software stability ? A big question mark. ConsPros

IP-over-OTN: Pros and Cons Reconfigurable optical backbone provides a flexible transport infrastructure Core OXC network can be shared with other service networks such as ATM, Frame Relay, and SONET/SDH private line services. Allows interconnection of IP routers in an arbitrary (logical) mesh topology. Not possible in architecture A since a typical CO/PoP has two, in some cases three, and in rare occasions four conduits connecting it to neighboring PoPs. Adding a reconfigurable optical backbone introduces an additional layer between the IP and DWDM layers and associated overhead. Traffic engineering occurs independently in two domains -- (i) the IP router network with its logical adjacencies spanning the OXC backbone, and (ii) the optical network which provisions physical lightpaths between edge IP routers. Could lead to inefficiency in traffic routing from a global perspective. ConsPros

Why Glass Through is not an Alternative? Removes the flexibility of dynamic switching between incoming and outgoing fibers at a PoP that comes with using a router or an OXC. Prevents organic growth of the network. Dynamic switching allows local capacity to be used to meet traffic demands between arbitrary PoPs. With glass through, bandwidth is not available at the link level but only at the segment level whose two end PoPs terminate glass through fiber paths. Does not allow intelligent packet processing or performance monitoring of transit traffic at a PoP.

Network Deployment Cost Analysis Analysis of the two architectures from an economic standpoint. Contrary to common wisdom, a reconfigurable optical layer can lead to substantial reduction in capital expenditure for networks of even moderate size. Critical observation: Amount of transit traffic at a PoP is much higher than the amount of add-drop traffic. Hence, a reconfigurable optical layer that uses OXC ports (instead of router ports) to route transit traffic will drive total network cost down so long as an OXC interface is marginally cheaper than a router interface. Savings increases rapidly with the number of nodes in the network and traffic demand between nodes.

Assumptions: Network Model Typical CO/PoP has two, in some cases three, and in rare occasions four conduits connecting it to neighboring PoPs. Average degree = 2.5. Routing uniform traffic (equal traffic demand between every pair of PoPs) on networks of increasing size. Two traffic demand scenarios: uniform demand of 2.5 Gbps (OC-48) and 5 Gbps between every pair of PoPs. Multiple routers/OXCs can be placed at each PoP to meet port requirements for routing traffic. Core OXC network provides full grooming of OC-192 ports into OC-48 tributaries. Transit traffic uses router ports in IP- over-WDM and OXC ports (only) in IP-over-OTN. Quadruple redundant configuration of IP routers at a PoP to improve reliability and perform load-balancing. Shortest-hop routing of lightpaths. IP routers have upto 64 ports and OXCs have upto 512 ports (in keeping with port counts of currently shipped products). With or without traffic restoration (diverse backup paths).

Assumptions:Pricing IP routers and OXCs have fixed costs and per-port costs for OC-48 and OC- 192 interfaces. Ballpark list prices for currently shipped products. IP router: fixed cost of $200K and per-port cost of $100K and $250K for OC-48 and OC-192 interfaces respectively. OXC: fixed cost of $1M and per-post cost of $25K and $100K for OC-48 and OC-192 interfaces respectively.

2.5 Gbps of Traffic between PoP Pairs Without Restoration Cross-over point at network size of about 18 nodes.

Cross-over point at network size of about 15 nodes. 5 Gbps of Traffic between PoP Pairs Without Restoration

% of Transit Traffic in the Network Without Restoration 75-85% of the total traffic is transit traffic for a network size of 50 PoPs.

Cross-over point at network size of less than 8 nodes. 2.5 Gbps of Traffic between PoP Pairs With Restoration

Cross-over point at network size of less than 4 nodes. 5 Gbps of Traffic between PoP Pairs With Restoration

% of Transit Traffic in the Network With Restoration 80-95% of the total traffic is transit traffic for a network size of 50 PoPs.

Results and Discussion Without restoration: Network cost breakeven point occurs at network sizes of 18 and 15 nodes for 2.5 Gbps and 5 Gbps of uniform traffic respectively. With restoration: IP-over-OTN has lower cost beyond a network size of 4-6 nodes. IP-over-OTN becomes increasingly attractive as amount of traffic and network size grows. Savings is much more when we consider traffic restoration. Amount of transit traffic in the network grows rapidly as network size increases. For example, without restoration, 75-85% of the total traffic is transit traffic for a network size of 50 PoPs, and with restoration, it is 80-95%. Carrying transit traffic over OXC ports (instead of router ports) drives network cost down so long as an OXC interface is marginally cheaper than a router interface.

Results and Discussion contd.... With traffic restoration, the economies of scale reaped from IP-over-OTN is further increased. Each primary path in a network has a diversely routed backup path. Transit port usage will increase substantially when we consider backup paths while the number of terminating ports remains unchanged.

Case for Restoration at Optical Layer Restoration in IP-over-WDM: Provided at the IP layer where backup paths consume router ports (like primary paths). Restoration in IP-over-OTN: Can be provided at the optical or IP layers. In the former case, router ports are not consumed on intermediate PoPs. Study shows substantial increase in savings for IP-over-OTN when restoration is taken into consideration. IP-over-OTN has lower cost beyond a network size of 4-6 nodes. As much as 80-95% of the total traffic is transit traffic for a network size of 50 PoPs.

Ongoing Investigations in IP/OTN Networks Can IP layer provide reliable service? How much Restoration is really required for services? Interaction of Routing Protocols with Optical Layer Restoration Optimal Routing with Topology of IP and Optical Layers And many more...