1. A Method for Coordinated Multi-Domain Traffic Pattern Analysis Presented by: Julio Ibarra, Ernesto Rubi, James Grace, Christian Rodriguez Center for Internet Augmented Research and Assessment Florida International University Miami, FL CENIC 08 - ‘Lightpath to the Stars’ March 10, 2008 Award # OCI 0734173

2. Outline Introduction Research Questions Previous Work Solution Findings Future Work 2

3. Western Hemisphere Research & Education Networks – Links Interconnecting Latin America (WHREN-LILA) 5-year NSF Cooperative Agreement Connectivity to Brazil is supported through a coalition effort through the WHREN-LILA projects –Florida International University (award #0441095) –Corporation for Education Network Initiatives in California (CENIC) –Project support from the Academic Network of Sao Paulo (award #2003/13708- 0) –CLARA, Latin America –CUDI, Mexico –RNP, Brazil –REUNA, Chile Links Interconnecting Latin America (LILA) –Improves U.S.-Latin America connectivity Western-Hemisphere Research and Education Networks (WHREN) –Coordinating body of providers and users –Leverage participants’ network resources –Enable collaborative science research and advance education 3 3

4. WHREN-LILA 2.5Gbps circuit + dark fiber segment U.S. landings in Miami and San Diego Latin America landing in Sao Paulo, Tijuana and Miami LILA links are important assets that –support U.S.-Latin America science and education research activities –Major research facilities supporting international science collaborations 4

5. Project Motivation IRNC program review recommendation to assess appropriate use of network assets Opportunity from the NSF to submit proposal for Research Experience for Undergraduates (REU) program –The REU program supports active research participation by undergraduate students in any of the areas of research funded by the NSF Respond to review recommendation by conducting a study on the possibility collecting netflow data on the LILA links 5

6. Acknowledgments This research is funded by the National Science Foundation, Research Experience for Undergraduates award OCI 0734173 CENIC and the Conference organizers WHREN-LILA, AMPATH infrastructure, CHEPREO, Global CyberBridges, science application support, education, outreach and community building efforts are made possible by funding and support from: –National Science Foundation (NSF) awards OCI-0441095, MPS- 0312038, OISE-0549456, OCI-0537464, OCI 0636031, IIS 0646144, OISE 0715489,, OISE 0742675 –Florida International University –Latin American Research and Education community –The many national and international collaborators who support our efforts 6

7. Previous work – Design/Implementation NSF (STI): Research Experience for Undergraduates Award No. 331112 NetFlow based network monitoring (implemented): –Built-in historical component –Platform independent analysis interface (MonALISA). Single AS view (20080) Cisco (NetFlow) and Juniper (cflowd) data exported to single Collector Pre-Processing of NetFlow data before exporting it to ApMon/MonALISA Emphasis on Integration / Interoperability: –Scalable/Distributed Monitoring platform (MonALISA / UDP ApMon) –Open-source traffic analysis tools (FlowTools / NetFlow)‏ –Limited understanding of network behaviour outside AS 20080. 7

8. Pending Inquiry Expand beyond Single Flow TCP data analysis Multiple Source Port/Destination Port Multiple NetFlow collectors using data from geographically distributed routers. Sampled NetFlow –Not enough storage for 1:1 view of packets. (IOS/CPU Concerns) –Issue: Whether reliable inferences can be drawn from sampled 1:100 NetFlow data. –Just what are you missing? –Burst type traffic –Some of the longer flows –1:100 of the longer flows? 8

9. Research Objectives Increase understanding of the traffic patterns across the LILA links Determine if there are differences in traffic flows from both ends of the link Assess reliability of sampled NetFlow data collected at the end points Detect anomalies or events that could be significant 9

10. Research Questions What are the differences in traffic flows at both ends of the link? How reliable is the sampled netflow data collected at both ends of the link? How can anomalies be detected from sampled data? 10

11. Solution Validate Accuracy of Sampled NetFlow Data Collect Data from Endpoints of LILA link ANSP (Sao Paulo, Brazil) Correlate Data from Each Endpoint Miami, US and Sao Paulo, Brazil Draw Conclusions from Correlated Data 11

12. Path Representation SPRACE Cisco 7609 ANSP WHREN-LILA East (2.4Gb/s) AMPATH ultralog.ampath.netdavinci.ampath.net 12

13. Verification of Sampled Netflow Using tcpdump and trpr(U.S. Navy), we calculated and graphed the data transfer rate. We then compared these results to the sampled octet count from netflow. Each graph represents the transfer rate at measured from both sides of link. 13

14. Data Collection from each Endpoint Collection of data from AMPATH network using Netflow flow- capture command. Collect sampled NetFlow data at 15 minute intervals on a 1:100 random sampling rate Capture data from the collector to a local box via flow-capture command Store captured data in a file for correlation. Collection of data from ANSP network using open sourceTCPDump. Run TCPDump and collect packets coming from AMPATH. This data is a 1:1 sampling. Store TCPDump data in a local file for correlation. 14 Collection of data from AMPATH network using Cisco Netflow’s flow capture command. Collect sampled netflow data at 15 minute intervals on a 1:100 random sampling rate. Capture data from the collector to a local box via flow- capture command. Store captured data in a file for correlation. Collection of data from ANSP network using open sourceTCPDu mp. 1.) Run T C P D u m p a n d c o l l e c t p a c k e t s c o m i n g f r o m AMPAT H. T h i s d a t a i s a 1 : 1 s a m p l i n g. 2.) Store T C P D u m p d a t a i n a l o c a l f i l e f o r c o r r e l a t i o n.

15. Fast Data Transfer Transfers large amounts data over standard TCP streams. Resumes file transfer session without loss, when needed. Uses JAVA NIO library to create transfer. FDT must exist on two servers, one acts as an FDT client the other as an FDT server Proven extremely useful at CERN by setting the record for fastest TCP transfers. Server Example: java -jar fdt.jar [ OPTIONS ] Client Example: java -jar fdt.jar [ OPTIONS ] -c [file1...] -d 15

16. Fast Data Transfer Output example of 6 FDT flows to simulate Brazil T2 --> U.S. T1: FDT [ 0.8.7-200711141115 ] STARTED... READY 21/02 13:57:57Net In: 402.444 Mb/sAvg: 402.444 Mb/s 21/02 13:58:02Net In: 413.621 Mb/sAvg: 408.038 Mb/s 21/02 13:58:07Net In: 395.866 Mb/sAvg: 403.984 Mb/s 21/02 13:58:12Net In: 418.637 Mb/sAvg: 407.645 Mb/s 26.83% ( 58s ) 21/02 13:58:17Net In: 403.753 Mb/sAvg: 406.867 Mb/s 33.03% ( 53s ) 21/02 13:58:22Net In: 401.946 Mb/sAvg: 406.047 Mb/s 39.22% ( 48s ) ……………………. FDTWriterSession ( 2a665123-c278-4efe-854b-7389cbc900bd ) final stats: Started: Thu Feb 21 13:57:48 EST 2008 Ended: Thu Feb 21 14:00:22 EST 2008 TotalBytes: 4063883264 TotalNetworkBytes: 4063883264 Multiple FDT flows allow for a continuous rate of flow and consistent maximum use of bandwidth. FDT is limited by memory capacity of host because Java consumes many system resources. FDT was used to generate flows similar to flows between Tier2 in Brazil and Tier1 FermiLab in the U.S. 16

17. Correlate Data from each Endpoint Parsing NetFlow data with flow-cat, flow-nfilter, flow-print and awk flow-cat ft-v05.2008-02-11.194501-0500 | flow-nfilter -f filter -F foo | flow-print | awk '/198.32.252.3/ {print $6}' Concatenate flows Filter relevant dataPrint the dataOutput only sampled octet count Correlation of data is done using a variety software designed to interpret both netflow and pcap $./trpr input count exclude udp output Storing graph-able pcap data 17

18. Analysis of Correlated Data Interpret for detection of anomalies and network events A series of icmp echo packets are sent across LILA The Round-Trip-Time(RTT) is measured at different levels of link activity These correlations are plotted in a RTT vs. Sampled Octet Count graph RTT vs Time is also plotted. Comparing graphs allows the correlation of events happening on both ends of the link. 18

19. Findings ANSP (Brazil)AMPATH (Miami) SPRACE (Brazil) ICMP RTT Measurements over time to SPRACE, ANSP and AMPATH Effects of Anomalous behavior at SPRACE (Brazil) seen locally at Cisco 7609 (Miami). 108 ms average RTT measured from Miami to the ANSP router and server at SPRACE, both at Sao Paulo Graphs show variation from the mean from three different views as load increases from multiple flows Event occurring at 23:57:21 at SPRACE correlates to event occurring at Miami 19

20. Conclusions Cisco Netflow data is accurate when compared at both ends of the link with a sampling rate of 1:100 Using correlated data from sampled Netflow and ICMP flows, anomalous behaviour can be detected from one or more of the end points 20

21. Thank You 21