1 Resource Optimization in Hybrid Core Networks with 100G Links Malathi Veeraraghavan University of Virginia [Collaboration with Admela Jukan] Date: Sep. 28, 2009

Outline Problem statement Hybrid network architecture Traffic monitoring/characterization Traffic engineering Network engineering Traffic analysis & simulations DOE testbed Workplan 2

Big picture question As –link rates reach 100Gbps –switching capacities reach tens of Tb/s Do we need a new backbone network architecture? 3

Hybrid node architecture Unfolded view Input interfaces Output interfaces WDM switch 100Tbps Layer-3 (IP) router (1Tbps) Data plane SNMP MIB+agents Routing protocol Signaling (provisioning) protocol Administrative interface (CLI, TL1) Node controller Hybrid node Ethernet control-plane port Carrier Ethernet switch (10Tbps)

Hybrid network architecture 5 Shared single core pool of N fibers K circuits: IP-routed partition N-K: Dynamic-circuit partition Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Traffic engineering (TE) systemHybrid Network Engineering (NE) system “put the traffic where the bandwidth is”“put the bandwidth where the traffic is” REF: Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking. Available: Modify routing metrics and/or write routing table entries Request dynamic circuit setup/release Traffic monitoring/ characterization system DOE-implemented control-plane software systems Obtain data REF

Traffic monitoring/characterization system 6 Shared single core pool of N fibers K circuits: IP-routed partition N-K: Dynamic-circuit partition Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Traffic engineering (TE) systemHybrid Network Engineering (NE) system Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking. Available: Traffic monitoring/ characterization system DOE-implemented control-plane software systems Traffic monitoring/characterization system Reads parameters necessary only for TE/NE applications Characterizes traffic matrix Not itself a general-purpose monitoring system such as PerfSONAR but could interface with such systems to obtain data

Hybrid Traffic Engineering system 7 Shared single core pool of N fibers K circuits: IP-routed partition N-K: Dynamic-circuit partition Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Traffic engineering (TE) systemHybrid Network Engineering (NE) system Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking. Available: Traffic monitoring/ characterization system DOE-implemented control-plane software systems Hybrid Traffic Engineering (TE) system Obtains data from Traffic monitoring/characterization system Computes optimal routes for load balancing Issues CLI commands to hybrid nodes to modify routing metrics and/or write routing table entries

Hybrid network engineering system 8 Shared single core pool of N fibers K circuits: IP-routed partition N-K: Dynamic-circuit partition Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Traffic engineering (TE) systemHybrid Network Engineering (NE) system Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking. Available: Traffic monitoring/ characterization system DOE-implemented control-plane software systems Hybrid Network Engineering (NE) system Obtains data from Traffic monitoring/characterization system and DOE-implemented control-plane software Determines if thresholds are crossed to trigger setup/release of dynamic circuits If triggered, sends request for dynamic circuit setup/release to DOE-implemented control-plane software Commands Hybrid TE system to make routing table updates

What is today’s backbone network architecture 9 Two separate networks (separate nodes; separate links) –IP-routed service (black links) –SDN for dynamic-circuit service (blue links) Summer 09 ESnet map: IP routerSDN switch

Nodes and wide-area links IP routed network –16 routers (MX960 or M320) –20 inter-city (WAN) links – 10GbE SDN network –17 SDN switches (MX 960) two in New York, two in Chicago –29 inter-city (WAN) links – 10 GbE Metro rings –SUNN, CHIC, NEWY 10

Future hybrid network: Part of the ESnet topology For simplicity, let us drop the WDM switch from our hybrid node and just have one connection-oriented switch (e.g., carrier Ethernet, MPLS, WDM, SONET) 11 ALBU DENV PNWG SUNN ELPA Connection-Oriented (CO) switch 100 GbE IP Router 10GbE LANL SNLA Hybrid node PNNL

Hybrid architecture 12 SDN switch IP router ALBU SDN switchIP router DENV CO switchIP router ALBU CO switchIP router DENV 100 GbE10 GbE NREL dynamic circuit SNLA 10 GbE 1 GbE SNLA 10 GbE LANL 100 GbE Current: SeparateFuture: Hybrid NREL 10 GbE LANL What rate?

Comparison Current architectureHybrid (future) architecture Separate inter-city (backbone) links – Two or more 10GbE One inter-city (backbone) link GbE Most enterprise access links terminate only on IP router – 10GbE or 1GbE (some on both IP router and SDN switch) Enterprise access links terminate only on CO switch – 10 GbE or 100 GbE Backbone links are 10GbE (same as some enterprise access links) Can we make IP router-to-router circuits of lower rate? 13 Link loads are low now; nevertheless 10Gb/s links are required to avoid backbone links from becoming bottlenecks

Questions Backbone router to backbone router circuits: –bandwidth-limited? not mandatory to limit circuit bandwidth if CO switch is MPLS or carrier Ethernet –bottleneck link implication on long-flow TCP throughput –if so, what rate? Access circuits (enterprise-to-backbone): –what rate? 14

IP datagram flow in future hybrid network 15 ALBU DENV PNWG SUNN ELPA Connection-Oriented (CO) switch 100 GbE IP Router 10GbE LANL SNLA Hybrid node PNNL

Implication Packet forwarding: –CO switches handle packets at intermediate nodes –Backbone IP routers involved only at source and destination PoPs –“Cut-through” paths By reducing number of packets handled by IP router: –its’ switching capacity can be smaller –access and backbone circuit rates between access/backbone routers can be lower –low rates will not affect short flows, but will impact long flows –can long flows can be redirected to CO network? 16

Hybrid architecture feasibility study Traffic analysis Simulations –to enable planning of node size (port speeds, switching capacities) 17

Traffic analysis Analyze Netflow data –characterize “short” and “long” flows flow inter-arrival time distributions flow duration distributions –need to characterize lengths of silences between long flows generated by an enterprise –determine what applications generate “long” flows 18

Simulations Use Netflow data-validated input traffic model for ESnet Create a simulation model of an alternate “hybrid” ESnet, one in which inter-router links are of lower rates than in deployed network but used only for short flows Long flows are directed to CO network with dynamic circuit setup between enterprise routers Obtain performance metrics –short flows –long flows 19

SNMP traffic analysis Obtain SNMP link traces for backbone and access links Are some links that are not on many shortest paths loaded at lighter levels than others? Time-of-day dependencies? Correlations? Stationarity? 20

DOE-provided testbed Demonstrate operation of hybrid networks on DOE-provided testbed with 3 to 4 nodes. –Hybrid nodes (provided to us): IP router (layer 3) + SDN switch (layer 2.5, 2 or 1) DOE-implemented control-/management-plane software –OSCARS, PerfSONAR –Implemented by us: Traffic characterization software Traffic/networking engineering management systems 21

Work plan Year 1: Architecture and Analysis Year 2: Algorithm design and software implementation Year 3: Prototyping on DOE-provided testbed 22

Requests to ESnet Netflow data Raw SNMP data files RIBs (BGP and IGP) and IGP metrics Can a full-mesh of MPLS tunnels - without bandwidth allocations – be created on existing ESnet? –Simplifies the process of generating traffic matrices (LSP SNMP data) Are there experimental PCs at PoPs for researcher use? –Virtual machines? –PCs connected to both IP router and SDN switch? –Run BWdetail (data-plane experiments) Is to be maintained? If so, the Public Access link gives error messages. Will Level2 access be useful to us? 23