Network Architectures OARTech Paul Schopis October 13, 2005

Topics TFN/OARnet background General MPLS Description What problem are we trying to solve anyway? Early Experiments at ITEC TFN implementation

OARnet Background Founded in 1987 as part of the Ohio Supercomputing Center 90+ higher ed member institutions Board of Regents funding OSTEER advisory council Internet2 GigaPOP

Third Frontier Network Phase 1: replace backbone with dark fiber Phase 2: connect 17 universities to network with dark fiber or gig circuits Phase 3: connect other universities and colleges Phase 4: connect other partners

Dark Fiber Acquisition RFP issued during Summer of 2002 Dark fiber was strongly preferred, but leased services considered Vendors who bid dark fiber were required to offer a minimum of a single pair of fiber over their network

Dark Fiber Acquisition Determined that leased lambdas were too expensive and not widely available Selected a bid from Spectrum Networks for single pair of fibers –American Electric Power (AEP) –Williams Communications (Wiltel) –American Fiber Systems

Spectrum We had various responses The providers in the Spectrum offer bid individually No price increase for using Spectrum as integrator SBC and others reported no bid bit desired to bid on future last mile –inter-lata issue

Dark Fiber Acquisition $4.6 M for 20 year IRUs $342K/yr for maintenance route miles Truewave, SMF-28, LEAF or Terra Light Fiber Aerial and buried

TFN Financing $21M investment Financing from Ohio State University –Loan for fiber ($7M) –Short-term financing ($2M) Financing from state capital budget ($8.5M) –Equipment –Last mile to 17 institutions

Community We desired to make this a true community owned network Committees with schools participating in decisions and recommendations

Equipment Cisco integrated solution (DWDM) –all of the amps, mux/demux etc. integrated Multi Service Transport Platform (MSTP) –ITU G.709 compliant Cisco routers (GSR 12000) and switches Juniper M7i routers

Last Mile RFP issued in Dec 2003 for last-mile connectivity to all higher education and K-12 sites OC3, gig circuits and10 gig circuits We did make contact with local fiber providers on backbone bid ex. Buckeye Telesys

General MPLS Description

Packet have a 20 bit label that routes it along a “Label Switched Path”. Values range from 0 to 1,048, through 15 are reserved for special uses. Some label ranges have special meanings for specific vendors.

General MPLS Description 0 IPv4 Explicit Null Label - No label stacking, must POP label 1 Router Alert Label - delivered to local router for local processing 2 IPv6 Explicit Null Label - Same rule as IPv4 except forwarded to IPv6 routing instance. 3 Implicit Null Label - Control protocol (LDP or RSVP) request for down stream router to POP Label

General MPLS Description Choosing the next hop can be thought of as the composition of two functions. The first function partitions the entire set of possible packets into a set of"Forwarding Equivalence Classes (FECs)". The second maps each FEC to a next hop. In many ways an IP prefix is a FEC IP routing protocols are the mechanisms to map IP FECs to a next hop.

General MPLS Description What are the advantages of MPLS?

General MPLS Description MPLS forwarding can be done by switches which are capable of doing label lookup and replacement, but are either not capable of analyzing the network layer headers, or are not capable of analyzing the network layer headers at adequate speed.

General MPLS Description Since a packet is assigned to a FEC when it enters the network,the ingress router may use, in determining the assignment, any information it has about the packet, even if that information cannot be gleaned from the network layer header. For example,packets arriving on different ports may be assigned to different FECs. Conventional forwarding, on the other hand,can only consider information which travels with the packet in the packet header.

General MPLS Description A packet that enters the network at a particular router can be labeled differently than the same packet entering the network at a different router, and as a result forwarding decisions that depend on the ingress router can be easily made. This cannot be done with conventional forwarding, since the identity of a packet's ingress router does not travel with the packet.

General MPLS Description Sometimes it is desirable to force a packet to follow a particular route which is explicitly chosen at or before the time the packet enters the network, rather than being chosen by the normal dynamic routing algorithm as the packet travels through the network. This may be done as a matter of policy,or to support traffic engineering. In conventional forwarding,this requires the packet to carry an encoding of its route along with it ("source routing"). In MPLS, a label can be used to represent the route, so that the identity of the explicit route need not be carried with the packet.

General MPLS Description Some routers analyze a packet's network layer header not merely to choose the packet's next hop, but also to determine a packet's"precedence" or "class of service". They may then apply different discard thresholds or scheduling disciplines to different packets.MPLS allows (but does not require) the precedence or class of service to be fully or partially inferred from the label. In this case, one may say that the label represents the combination of a FEC and a precedence or class of service.

What problem are we trying to solve anyway?

The Problem Goal create an Abilene Premium Service Need to create “Virtual Wire” ( Smells a lot like a light path) Need predictable bandwidth Need to meet DiffServ EF requirements Need to be able to signal request for resources Needed admission control

The Solution DiffServ Code Point Queuing mechanisms High Priority Policy on edge to mark and forward via high priority queue Admission control for LSP (MPLS Tunnels) via marked packets that conformed to requirements

The Solution LSPs anchored to WRED Queues on WAN side All CPE side used High Priority Tested across multiple BGP Domains Tested QPPB for discovery of QoS resources

The Solution Used RSVP to signal request for “sub- pool” reservation, e.g. guaranteed BW Resulted in primitives being incorporated into DSTE-MPLS Results used to write RFC 3270

AS 1 AS 3 AS 2 AS 3 AS 4

AS 1 AS 3 AS 2 AS 3 AS 4

AS 1 AS 3 AS 2 AS 3 AS 4

AS 1 AS 3 AS 2 AS 3 AS 4

TFN implementation

Needed to migrate to new network Needed to provide services such as multicast and IPv6 Needed to solve fish problem Executed test plan based on Abilene test plan

Legacy POP Design I1 I2 ATM I2 I1 ATM I1&I2 ATM POP Campus

Legacy POP Design I1 I2 ATM I2 I1 ATM I1&I2 ATM POP Campus BGP for Route diff

New OARnet Design Goals Reduce Costs Reduce Complexity Reduce Maintenance Fees Deliver Services

MPLS Requirements CPE device PE Provider Edge P Provider Core LSP Switching Router We can collapse P and PE to one device Need CPE for Label to IP binding I1 will be standard routing I2 will be Label Switched with BGP multihop to find correct path Must deliver advanced services to I2 community –IPv6, Multicast, Jumbo Frames etc.

New Architecture PE/P CPE GigE Aggregator CPE GigE POP Campuses

New Architecture PE/P CPE GigE Aggregator CPE GigE Campuses BGP Multihop LDP Exchange with Core BGP Multihop LDP Exchange with Core MPLS for I2 Routes LFIB IP for I1 Routes FIB Red = LDP tagged AS3112 AS600

New Architecture PE/P LR 1 GigE Aggregator CPE GigE Campuses AS3112 AS600 LR 2

Rate Cap Architecture PE/P CPE GigE Aggregator CPE GigE Campuses Red = I2 Cap Blue = Commodity Cap Green = Intra State Cap AS3112 AS600

Some Implementation Issues Had to come up with more robust naming convention –Old ALP1, SWALP1 Required DNS overhaul –Pseudo CILLY code CLMBN-R0, CLMBN-E0, CLMBN-O1, CLMBN- OT1

Questions?