1 Traffic Engineering of High-Rate Large-sized Flows Acknowledgment: UVA work is supported by DOE ASCR grants DE-SC and DE-SC , and NSF grants, OCI , OCI , and CNS , and ESnet work is supported by DOE grant DE-AC02-05CH11231 Tian Jin, Chris Tracy, Malathi Veeraraghavan, Zhenzhen Yan University of Virginia and ESnet July 8-11, 2013

Outline Problem statement & Motivation –Example of ESnet measured load –Adverse effects of “alpha flows” Hybrid Network Traffic Engineering System (HNTES) HNTES evaluation –NetFlow data collection –Effectiveness –Afflicted-flow packet percentage 2

Problem statement Flows generated by high-rate large-sized file transfers are called alpha flows –thresholds used in this paper: 1 GB in  1 min Previous work shows that alpha flows –are the cause of burstiness of IP traffic Experiment shows adverse effects of alpha flows on real-time A/V flows Problem: How can a provider identify such alpha flows within their network and direct them to separate QoS-controlled VCs? 3

Motivation: ESnet4 Core network for US Dept. of Energy Labs StarLight MAN LAN (32 A of A) PNNL FNL ORNL LLNL GA BNL LANL IP router Lab Optical node SDN router Lab Link MAN NLR 10G 30/40/50G SDN IP Steve Cotter, Chin Guok, Joe Metzger, Bill Johnston Brookhaven National Laboratory

Traffic surges on ESnet interface 5 Link rate: 10 Gbps Outgoing traffic Incoming traffic 9 Gbps Jan. 12, 2013

Motivation: Adverse effects of alpha flows Used DOE 100G testbed Hosts: high-performance diskpts 6 BNL NEWY ping flow (delay-sensitive) TCP (alpha) flow UDP flow (background) buffer buildups

Impact of alpha flows on real-time flows 7 Impact on ping flow delay –significant in 1-queue configuration –negligible in 2-queue configuration Need separate virtual queue for alpha flow packets Pings: 1 per sec Delay: 60 ms in 1-queue case Delay: 2.1 ms in 2-queue case UDP flow TCP flow 3 Gbps 6 Gbps

Outline Problem statement & Motivation  Hybrid Network Traffic Engineering System (HNTES) HNTES evaluation –NetFlow data collection –Effectiveness –Afflicted-flow packet percentage 8

Hybrid network traffic engineering system (HNTES) - Intradomain identification/redirection of alpha flows 9 Three steps –Analysis of NetFlow reports from ingress routers to identify address prefixes of completed alpha flows –IDC creates L3 circuits between ingress-egress router pairs and configures QoS –IDC sets firewall filters to direct future alpha flows with matching address prefixes to L3 circuits Aging parameter (A): age out rules corresponding to prefixes for which no alpha flows have been observed

Outline Problem statement & Motivation Hybrid Network Traffic Engineering System (HNTES) HNTES evaluation –NetFlow data collection –Effectiveness –Afflicted-flow packet percentage 10

Data collection for HNTES evaluation: NetFlow data from 4 routers were collected for 7 months (214 days) 11 router-1 & router-2: provider-edge (PE) routers router-3: core router (REN peering) router-4: core router (commercial peering) OP: observation point

Effectiveness Analysis Two types of effectiveness –Cumulative effectiveness (C i ): percent of alpha bytes (bytes reported in alpha NetFlow reports) that would have been redirected in period (1,i) –Daily effectiveness (E i ): percent of alpha bytes that would have been redirected on day i Choose aging parameter for: –High effectiveness –Stability in firewall-filter size 12

Aging parameter: tradeoff effectiveness with size of firewall filter graphs for router 1 (similar for other routers) 30 days is good compromise for aging parameter 13 Firewall filter size stable with aging parameter 30 Cumulative effectiveness > 90%

Cumulative effectiveness (/24) 14 Provider edge routers (single customers) Peering routers (router-3: REN; router-4: commercial) Why is cumulative effectivness lower for peering routers, esp. router-4? Boxplots for 214 values each router-1 omitted as it is similar to router-2 Cumulative effectiveness

Effectiveness comparisons 15 Obs. 1: higher effectiveness for /24 than for /32 Obs. 2: higher effectiveness for router-1 and router-2 than for router-3 and router-4 Obs. 3: fewer alpha prefix IDs for router-3 and router-4

Explanations 16 Obs. 1 : data-transfer node clusters are typically located in the same /24 subnet; thus, repetition is greater with /24 than /32 Obs. 2 and obs. 3: Higher effectiveness for routers 1 & 2: downloads from supercomputing facilities are repetitive (a scientist accesses the same data transfer nodes) Lower effectiveness for routers 3 & 4: fewer uploads to DoE labs than downloads from DOE labs expect few, if any, scientific data transfers from commerical peers (router-4)

Outline Problem statement & Motivation Hybrid Network Traffic Engineering System (HNTES) HNTES evaluation –NetFlow data collection –Effectiveness –Afflicted-flow packet percentage 17

Afflicted-flow packets B: set of non-alpha NetFlow reports for flows that share alpha prefix IDs Divide B into four subsets in sequence –C: non-alpha reports of alpha flows –D  B-C: data-transfer reports (heuristic) –W  B-C-D: well-known ports –L: leftover = B-C-D-W Afflicted flows: W+L 18

Afflicted-flow packets Tradeoff: /24 vs /32 –/32 has lower effectiveness: large % of afflicted-flow packets will be impacted when an alpha flow is not redirected –/24 has higher afflicted-flow packet percentage: small % of afflicted-flow packets are adversely impacted Recommend /24 address prefixes for firewall filters 19 Percentage of afflicted-flow packets in samples of beta-flow (non- alpha flow) packets; across the 214-day period

Conclusions Hypothesis: Most high-speed data transfer nodes have static IP addresses, and alpha flows are created repeatedly between the same source-destination subnets –Validated for flows generated by dataset downloads as observed at edge routers HNTES solution of determining src-dest address prefixes of completed alpha flows & using these prefixes to set firewall filters for future alpha-flow redirection is effective for downloads from DOE labs Less effective for uploads esp. from commercial peering links – But alpha-flow causing uploads are fewer 20