1. 1 Resource Optimization in Hybrid Core Networks with 100G Links Malathi Veeraraghavan University of Virginia [Collaboration with Admela Jukan] Date: Sep. 28, 2009

2. Outline Problem statement Hybrid network architecture Traffic monitoring/characterization Traffic engineering Network engineering Traffic analysis & simulations DOE testbed Workplan 2

3. Big picture question As –link rates reach 100Gbps –switching capacities reach tens of Tb/s Do we need a new backbone network architecture? 3

4. Hybrid node architecture Unfolded view Input interfaces Output interfaces WDM switch 100Tbps Layer-3 (IP) router (1Tbps) Data plane SNMP MIB+agents Routing protocol Signaling (provisioning) protocol Administrative interface (CLI, TL1) Node controller Hybrid node Ethernet control-plane port Carrier Ethernet switch (10Tbps)

5. Hybrid network architecture 5 Shared single core pool of N fibers K circuits: IP-routed partition N-K: Dynamic-circuit partition Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Traffic engineering (TE) systemHybrid Network Engineering (NE) system “put the traffic where the bandwidth is”“put the bandwidth where the traffic is” REF: Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking. Available: http://networks.cs.ucdavis.edu/mukherje/US-EU-wksp-June05-Final-Report.pdf Modify routing metrics and/or write routing table entries Request dynamic circuit setup/release Traffic monitoring/ characterization system DOE-implemented control-plane software systems Obtain data REF

6. Traffic monitoring/characterization system 6 Shared single core pool of N fibers K circuits: IP-routed partition N-K: Dynamic-circuit partition Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Traffic engineering (TE) systemHybrid Network Engineering (NE) system Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking. Available: http://networks.cs.ucdavis.edu/mukherje/US-EU-wksp-June05-Final-Report.pdf Traffic monitoring/ characterization system DOE-implemented control-plane software systems Traffic monitoring/characterization system Reads parameters necessary only for TE/NE applications Characterizes traffic matrix Not itself a general-purpose monitoring system such as PerfSONAR but could interface with such systems to obtain data

7. Hybrid Traffic Engineering system 7 Shared single core pool of N fibers K circuits: IP-routed partition N-K: Dynamic-circuit partition Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Traffic engineering (TE) systemHybrid Network Engineering (NE) system Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking. Available: http://networks.cs.ucdavis.edu/mukherje/US-EU-wksp-June05-Final-Report.pdf Traffic monitoring/ characterization system DOE-implemented control-plane software systems Hybrid Traffic Engineering (TE) system Obtains data from Traffic monitoring/characterization system Computes optimal routes for load balancing Issues CLI commands to hybrid nodes to modify routing metrics and/or write routing table entries

8. Hybrid network engineering system 8 Shared single core pool of N fibers K circuits: IP-routed partition N-K: Dynamic-circuit partition Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Node Hybrid Traffic engineering (TE) systemHybrid Network Engineering (NE) system Report of US/EU Workshop on Key Issues and Grand Challenges in Optical Networking. Available: http://networks.cs.ucdavis.edu/mukherje/US-EU-wksp-June05-Final-Report.pdf Traffic monitoring/ characterization system DOE-implemented control-plane software systems Hybrid Network Engineering (NE) system Obtains data from Traffic monitoring/characterization system and DOE-implemented control-plane software Determines if thresholds are crossed to trigger setup/release of dynamic circuits If triggered, sends request for dynamic circuit setup/release to DOE-implemented control-plane software Commands Hybrid TE system to make routing table updates

9. What is today’s backbone network architecture 9 Two separate networks (separate nodes; separate links) –IP-routed service (black links) –SDN for dynamic-circuit service (blue links) Summer 09 ESnet map: http://www.es.net/pub/maps/current.pdf IP routerSDN switch

10. Nodes and wide-area links IP routed network –16 routers (MX960 or M320) –20 inter-city (WAN) links – 10GbE SDN network –17 SDN switches (MX 960) two in New York, two in Chicago –29 inter-city (WAN) links – 10 GbE Metro rings –SUNN, CHIC, NEWY 10

11. Future hybrid network: Part of the ESnet topology For simplicity, let us drop the WDM switch from our hybrid node and just have one connection-oriented switch (e.g., carrier Ethernet, MPLS, WDM, SONET) 11 ALBU DENV PNWG SUNN ELPA Connection-Oriented (CO) switch 100 GbE IP Router 10GbE LANL SNLA Hybrid node PNNL

12. Hybrid architecture 12 SDN switch IP router ALBU SDN switchIP router DENV CO switchIP router ALBU CO switchIP router DENV 100 GbE10 GbE NREL dynamic circuit SNLA 10 GbE 1 GbE SNLA 10 GbE LANL 100 GbE Current: SeparateFuture: Hybrid NREL 10 GbE LANL What rate?

13. Comparison Current architectureHybrid (future) architecture Separate inter-city (backbone) links – Two or more 10GbE One inter-city (backbone) link - 100 GbE Most enterprise access links terminate only on IP router – 10GbE or 1GbE (some on both IP router and SDN switch) Enterprise access links terminate only on CO switch – 10 GbE or 100 GbE Backbone links are 10GbE (same as some enterprise access links) Can we make IP router-to-router circuits of lower rate? 13 Link loads are low now; nevertheless 10Gb/s links are required to avoid backbone links from becoming bottlenecks

14. Questions Backbone router to backbone router circuits: –bandwidth-limited? not mandatory to limit circuit bandwidth if CO switch is MPLS or carrier Ethernet –bottleneck link implication on long-flow TCP throughput –if so, what rate? Access circuits (enterprise-to-backbone): –what rate? 14

15. IP datagram flow in future hybrid network 15 ALBU DENV PNWG SUNN ELPA Connection-Oriented (CO) switch 100 GbE IP Router 10GbE LANL SNLA Hybrid node PNNL

16. Implication Packet forwarding: –CO switches handle packets at intermediate nodes –Backbone IP routers involved only at source and destination PoPs –“Cut-through” paths By reducing number of packets handled by IP router: –its’ switching capacity can be smaller –access and backbone circuit rates between access/backbone routers can be lower –low rates will not affect short flows, but will impact long flows –can long flows can be redirected to CO network? 16

17. Hybrid architecture feasibility study Traffic analysis Simulations –to enable planning of node size (port speeds, switching capacities) 17

18. Traffic analysis Analyze Netflow data –characterize “short” and “long” flows flow inter-arrival time distributions flow duration distributions –need to characterize lengths of silences between long flows generated by an enterprise –determine what applications generate “long” flows 18

19. Simulations Use Netflow data-validated input traffic model for ESnet Create a simulation model of an alternate “hybrid” ESnet, one in which inter-router links are of lower rates than in deployed network but used only for short flows Long flows are directed to CO network with dynamic circuit setup between enterprise routers Obtain performance metrics –short flows –long flows 19

20. SNMP traffic analysis Obtain SNMP link traces for backbone and access links Are some links that are not on many shortest paths loaded at lighter levels than others? Time-of-day dependencies? Correlations? Stationarity? 20

21. DOE-provided testbed Demonstrate operation of hybrid networks on DOE-provided testbed with 3 to 4 nodes. –Hybrid nodes (provided to us): IP router (layer 3) + SDN switch (layer 2.5, 2 or 1) DOE-implemented control-/management-plane software –OSCARS, PerfSONAR –Implemented by us: Traffic characterization software Traffic/networking engineering management systems 21

22. Work plan Year 1: Architecture and Analysis Year 2: Algorithm design and software implementation Year 3: Prototyping on DOE-provided testbed 22

23. Requests to ESnet Netflow data Raw SNMP data files RIBs (BGP and IGP) and IGP metrics Can a full-mesh of MPLS tunnels - without bandwidth allocations – be created on existing ESnet? –Simplifies the process of generating traffic matrices (LSP SNMP data) Are there experimental PCs at PoPs for researcher use? –Virtual machines? –PCs connected to both IP router and SDN switch? –Run BWdetail (data-plane experiments) Is http://netinfo1.es.net to be maintained? If so, the Public Access link gives error messages. Will Level2 access be useful to us? 23