1. Past research work and research work in progress on elephant flows
Presented by
Sourav Maji
Advisor: Prof. Malathi Veeraraghavan
Work was done by present and past students of HSN group at UVa in collaboration with ESnet and Reservoir Labs
Thanks to the NSF for grants , , , , and the US DOE ASCR for grant DE-SC

2. Current group members at University of Virginia
Prof. Malathi Veeraraghavan
Sherry Wang
Fatemah Alali
Gholam Reza Rahimi 
Sourav Maji
Guthrie Ober Alexander

3. Outline Background Previous work on elephant flows
Traffic Engineering of High-Rate Large-Sized Flows
Characterization of high-rate large-sized flows
A measurement-based study of big-data movement
A novel Internet access service with online traffic engineering of elephant flows
Research work in progress on elephant flows
Other research work in High Speed Network group at University of Virginia

4. Energy Sciences (ESnet) ESnet offers network services to DOE labs (sites)
ALCF
NERSC
OLCF
NERSC: National Energy Research Scientific Computing Center
ALCF: Argonne Leadership Computing Facility
OLCF: Oak Ridge Leadership Computing Facility
Image: Courtesy: DOE: US Department of Energy

5. Sudden spikes on a 10 Gb/s ESnet link: cause?
10 Gbps link
Outgoing traffic
SNMP link utilization (Gbps)
Incoming traffic
Jan. 16, 2013
NetFlow analysis (packet header samples)
Shows these bursts are from 1 or 2 elephant flows
Elephant flows can cause packet losses of other flows and result in increased latency for delay sensitive flows

6. Traffic Engineering of High-Rate Large-Sized Flows
Goal was to design a hybrid network traffic engineering system (HNTES)
The system gathered NetFlow reports from routers, analyze these reports to identify elephant-flow source/destination address prefixes
HNTES design was evaluated using NetFlow reports collected over a 7-month period from four ESnet routers
Our analysis showed that if HNTES was deployed then 90% of elephant flow bytes could be redirected
Jin, Tian, et al. "Traffic engineering of high-rate large-sized flows." High Performance Switching and Routing (HPSR), 2013 IEEE 14th International Conference on. IEEE, 2013.

7. Characterization of high-rate large-sized flows
We described an algorithm for characterizing the size, duration, average rate, and frequency of such flows, from NetFlow reports
The algorithm can characterize flows as a part of a network management system for research-and-education network providers whose customers move large scientific datasets.
Flows moving datasets as large as 811 GB and at rates as high as 5.7 Gbps were observed
Of 1596 repeated elephant flows created by one pair showed considerable variance, with minimum rate of 100 Mbps, maximum rate of 536 Mbps
Jin, Tian, Chris Tracy, and Malathi Veeraraghavan. "Characterization of high-rate large-sized flows." Communications and Networking (BlackSeaCom), 2014 IEEE International Black Sea Conference on. IEEE, 2014.

8. Elephant flow (Alpha Flow visualization software)

9. Parallel flowsets

10. A measurement-based study of big-data movement
To accurately characterize big-data movement, it is important to reconstruct parallel flowsets from traffic measurements
Developed an algorithm to reconstruct individual elephant flows from the NetFlow records
The top 1% of flowset sizes were in the hundreds of GBs to low TBs range
95% of flowsets had rates less than 2.5 Gbps, and 99% of flowsets had durations shorter than 4 hours
These findings are useful for network planning, traffic engineering, and for improving user performance
Addanki, Ranjana, et al. "A measurement-based study of big-data movement." Networks and Communications (EuCNC), 2015 European Conference on. IEEE, 2015.

11. A novel Internet access service with online traffic engineering of elephant flows
The effectiveness of the offline HNTES solution was 67% in a particular case that could be addressed by online elephant flow identification
We design a high-speed Elephant Flow Traffic Engineering System (EFTES) that monitors access-link traffic and identifies elephant flows in real time
Upon identification of an elephant flow it redirects the elephant flow to a separate low priority queue
The EFTES algorithms were implemented in an innovative high-speed, multicore system called R-Scope.
EFTES could handle a 1-min traffic trace that contained 1.5 M flows with 8 cores.
Work is submitted in TNSM

12. Research work in progress (work 1)
Cost of classifying flows as elephants in real time
Address the problem of packets arriving out-of-sequence on redirection
Another question is to determine the cost of redirection for low-rate elephant flows versus high-rate elephant flows.
The EFTES identifies elephant flows based on a rate and duration threshold.
The research work is to determine through extensive simulations an algorithm to determining these thresholds

13. Research work in progress (work 2)
Throughput performance monitoring as a network service

14. Other research work in UVa HSN group
A Cross-Layer Multicast-Push Unicast-Pull (MPUP) Architecture for Reliable File-Stream Distribution
An Industrial Robotics Application with Cloud Computing and High- Speed Networking
Optical Switch in the Middle (OSM) Architecture for DCNs with Hadoop Adaptations
A WAN Scavenger Service for High-Speed Inter-Datacenter Communications
QoS-guaranteed EEE