2. -1- Wise* TrafView Wise * TrafView ETRI’s Content-aware Internet Application Traffic Measurement and Analysis System APAN Network Technology WS January 29, 2004 IP Networking Technology Team, ETRI {jungsp, chunghs, choits, tsjeong}@etri.re.kr

3. -2- Wise* TrafView Contents ▣ Current Internet Application Traffic Characteristics ▣ Wise* TrafView : Our Approach ▣ Wise* TrafView : Implementation and Deployment Experiences ▣ Summary

4. -3- Wise* TrafView Measurement Application Areas ▣ Network Problem Determination and Analysis ▣ Traffic Report Generation ▣ Intrusion & Hacking Attack (e.g., DoS, DDoS) Detection ▣ Service Level Monitoring (SLM) ▣ Network Planning ▣ Usage-based Billing/Accounting (both between SPs and SP- and-Customer) ▣ Customer Relationship Management (CRM) ▣ Marketing

5. -4- Wise* TrafView 2. CURRENT INTERNET APPLICATION TRAFFIC CHARACTERISTICS

6. -5- Wise* TrafView Current Internet Traffic Characteristics ▣ High-speed networks (Mbps  Gbps  Tbps) ▣ High-volume traffic ▣ Variety of Applications ◈ Streaming media (Windows Media, Real Media, Quicktime) ◈ P2P traffic ◈ Network Games ◈ Network Security Attacks ◈ Etc.

7. -6- Wise* TrafView Application Recognition(1) ▣ Limitations of port-based recognition ◈ The port database maintained by IANA doesn’t reflect the real-world situation –Most newer applications simply do not register their ports –Sometimes they even invade well-known port area to pass thorough firewalls ◈ Most bandwidth hogs, nowadays, dynamically allocate ports –They are not linked up with any fixed ports!

8. -7- Wise* TrafView Port/Application Port-based Accounting Contents-aware Accounting 80/HTTP67 GB 59.1 GB (11.8% reduced) 21/FTP_CTRL0.29 GB0.28 GB 20/FTP_DATA43 GB42 GB ?/FTP_DATA_PASSIVEn/a 6 GB (14.3% of FTP_DATA, 2% of the total volume) 5003/?692 MB HTTP: 13.2 MB BUGS_MUSIC: 420.8 MB EDONKEY: 172.3 MB etc.: 85.7 MB PosTech Traffic Breakdown - PosTech Campus Network (24h sum in May, 304GB total volume) Application Recognition(2) : Trend in Internet Application Traffic Characteristics

9. -8- Wise* TrafView Application Recognition(3) ▣ Many applications require to be identified by payload inspection ▣ Why is payload inspection necessary? ◈ Several applications can use the same port number ◈ Identification error can be occurred by ephemeral port number ◈ Some applications can use a dynamic port number ◈ Etc.

10. -9- Wise* TrafView ▣ Application example : Passive FTP Application Recognition(4) client.1302 server.21 (FTP_CTRL_REQ) server.21 (FTP_CTRL_REP) client.1302 client.1303 server.20 (FTP_DATA_DOWN) server.20 (FTP_DATA_UP) server.49152 (FTP_DATA_PSV_UP) client.1306 server.49152 (FTP_DATA_PSV_DOWN) % ls % passive % get wmggw.mp3 % quit % ftp server 49152

11. -10- Wise* TrafView Why Port-based Approach is not enough? ▣ Non-flow based measurement ◈ Not enough for the above requirements ▣ Typical Flow-based Measurement (like NetFlow TM, cflowd, LFAP) ◈ Typically a flow is defined as a set of packets passing an observation point in the network during a certain time interval and having a set of common properties ◈ 5-tuple packet header fields are used for this ◈ New applications such as P2P, streaming and network games have characteristics of dynamic port allocation ▣ More Detailed Analysis is needed!! ◈ Typical Flow-based Measurement is not enough ◈ Need more detailed analysis depending on applications –It may require content filtering

12. -11- Wise* TrafView 3. Wise* TrafView : OUR APPROACH

13. -12- Wise* TrafView Motivation ▣ Develop precise Internet application traffic measurement and analysis system ◈ Precise application analysis ◈ Passive flow-based measurement ◈ Sub-transaction(flow) level detailed application analysis ◈ Pseudo-realtime analysis ◈ No loss capture and analysis ◈ No sampling but capturing all ◈ For various Internet measurement purposes

14. -13- Wise* TrafView Flow Concept ▣ A “flow” is ◈ a sequence of packets whose are all identical ▣ Why flow? ◈ The size of entire raw packet streams for a given unit time are prohibitively enormous to be analyzed in time ◈ Each individual packets in a flow contain duplicate information ◈ Packets in the same flow are correlated; we can identify more packets which were previously categorized as unknown application a packet a distinctive signature of application “X” a flow generated by application “X” Now, these pkts can also be identified as “X”

15. -14- Wise* TrafView Internet Application Classification ▣ Type S: Simple Application Type ◈ for an application which uses a well-known port number or which uses a registered port number but is popularly used ◈ Applications : WWW, FTP, SMTP, BGP, etc. ▣ Type P: Payload Application Type ◈ for an application which uses a registered port number but requires payload inspections for precise classification ◈ Applications : HTTP_ALT(8080,8081,9000), MSNMessenger(6891-6900), KAZZA(1214), … ▣ Type R: Reverse Application Type ◈ for an application which uses a registered but requires comparison with a correlated reverse flow for the precise classification ◈ Applications : eDonkey down, WINMX down, GuruGuru BBS(9999)… ▣ Type C: Co-related Application Type ◈ for an application which uses a dynamic port number assignment ◈ Applications : Passive FTP, RTSP, Windows Streaming, …

16. -15- Wise* TrafView Capture Agent Analysis Server Database GUI... NIC IPCAP Card... NIC IPCAP Card splitter flow and packet records (NFS) recognition and analysis results (ODBC) ARCL Config-File... System Architecture Overview

17. -16- Wise* TrafView Agent : Generating Flow & Packet Records ▣ Carries on simple filtering and signature matching functions ▣ Generates flow records & packet records ◈ Flow record –For flow information –Fields : IP addr, port, protocol, flow duration, packets, bytes, … ◈ Packet record –for individual packet –Fields : timestamp, TOS, TTL, TCP flags, payload, … –Important for analysis server’s precise application identification ◈ This procedure aggregates and organizes the traffic information and reduces the amount of traffic volume transferred to the server

18. -17- Wise* TrafView Analysis Server : Enhanced Application Recognition ▣ Wise* TrafView utilizes some enhanced proprietary recognition mechanisms in a comprehensive way ◈ Application specific signature matching, ◈ temporal and spatial flow correlation, ◈ dynamic port recognition and utilization, and ◈ some heuristics ▣ Not only capable of discriminating applications, but also their sub-flows ◈ e.g., HTTP  HTTP_REQ, HTTP_REP, HTTP_REQACK, etc.

19. -18- Wise* TrafView Analysis Server : AS and Country Mapping ▣ Identifying flow sources and destinations ◈ Both source and destination IP address of a flow are mapped to ASes and finally to country codes ◈ This helps to locate the source and the sink of a flow –enables discrimination among transit, inbound, and outbound traffic flows

20. -19- Wise* TrafView Application Recognition Configuration Language (ARCL) ▣ Configurability and Adaptability ▣ Why adaptability so important? ◈ The highly frequent nature of Internet applications’ appearance and disappearance ◈ Swift mutation of applications ◈ Localization of the use patterns of applications ▣ Wise* TrafView copes with the problem by introducing ARCL (Application Recognition Configuration Language) ▣ By taking advantage of ARCL, Wise* TrafView ◈ doesn’t need to be re-built or re-installed by any module for extending or modifying recognition coverage; editing the configuration in ARCL and re-enforcing suffices

21. -20- Wise* TrafView Config-file by ARCL application WWW { port_rep_name HTTP port 80 protocol TCP{ // S type decision_group HTTP_REQ_REP_ACK { src_port >= 1024 dst_port == 80 } decision_group HTTP_REP_REQ_ACK { src_port == 80 dst_port >= 1024 } port_rep_name HTTP_ALT port 8080 protocol TCP{ // P type src_disc_pattern=="HTTP" in pkt 0-2 at byte 0 - 4 ( dst_disc_pattern=="GET" in pkt 0-3 at byte 0 - 10 || dst_disc_pattern=="POST" in pkt 0-3 at byte 0 - 10 ) decision_group HTTP_ALT_REQ_REP_ACK { src_port >= 1024 dst_port == 8080 } decision_group HTTP_ALT_REP_REQ_ACK { src_port == 8080 dst_port >= 1024 } application EDONKEY { // R type port_rep_name EDONKEY_DOWN port 4662 protocol TCP{ dst_disc_pattern=="0xe33d000000" in pkt 2-3 at byte 0 - 4 decision_group EDONKEY_DOWN_REQ_REP_ACK { src_port >= 1024 dst_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 } decision_group EDONKEY_DOWN_REP_REQ_ACK { src_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 dst_port >= 1024 } …… } application FTP { // C type port_rep_name FTP port 21 protocol TCP{ src_ref_pattern=="r/227 Entering Passive Mode \(\d{1,3},\d{1,3},\d{1,3},\d{1,3},(\d{1,4}),(\d{1,4})\)/$src_port = atoi($1)*1024 + atoi($2)" in pkt any at byte 0-35 induce FTP_DOWN_P decision_group FTP_REQ_REP_ACK { src_port >= 1024 dst_port == 21 } decision_group FTP_REP_REQ_ACK { src_port == 21 dst_port >= 1024 }

22. -21- Wise* TrafView 4. Wise* TrafView : IMPLEMENTATION & DEPLOYMENT EXPERIENCES

23. -22- Wise* TrafView Deployment Experiences ▣ ETRINet ◈ Link speed : 100Mbps FastEthernet, using libpcap ◈ Traffic volume : 70Mbps ◈ Period : May 2003 – Current ◈ Analysis result : S(52.83%), P(9.99%), R(2.38%), C(4.92%), Unknown(28.88%) ▣ Postech ◈ Link speed : 1Gbps Ethernet, using libpcap ◈ Traffic Volume : 60 – 70Mbps ◈ Period : May 2003(1week) ▣ Univ. of Andong ◈ Link speed : FastEthernet, using capturing card developed by ETRI ◈ Traffic volume : 60-70Mbps ◈ Period : Oct. 2003 - Current ▣ Other experiences ◈ Deployment on the International link of one of Korean Internet Exchange point using OC-3 POS card developed by ETRI

24. -23- Wise* TrafView Port/Application Port-based Accounting (A) Contents-aware Accounting (B) Accuracy (A/B) 80/HTTP67 GB 59.1 GB (11.8% reduced) 0.882/1.0 21/FTP_CTRL0.29 GB0.28 GB0.965/1.0 20/FTP_DATA43 GB42 GB0.977/1.0 ?/FTP_DATA_PAS SIVE n/a 6 GB (14.3% of FTP_DATA, 2% of the total volume) 0.0/1.0 5003/?692 MB HTTP: 13.2 MB0.0/1.0 BUGS_MUSIC: 420.8 MB0.0/1.0 EDONKEY: 172.3 MB0.0/1.0 etc.: 85.7 MB0.0/1.0 - PosTech Campus Network (24h sum in May, 304GB total volume) PosTech Traffic Analysis Result

25. -24- Wise* TrafView ▣ Hardware ◈ For lower speed links (<= 622Mbps) –Capture agent –high performance PC: Zeon 2.4GHz * 2 + CPU, 2GB+ RAM –Analysis server –high performance PC: Zeon 2.8GHz * 2 + CPU, 1GB+ RAM, 100GB+ HDD ◈ For Higher speed links ( > 1 Gbps, under developing) –Clustered capture system –Hardwired logic for supporting wire-speed processing ▣ Software ◈ Capture agent –Linux ◈ Analysis server –Linux, MySQL System Spec.(1)

26. -25- Wise* TrafView System Spec.(2) ▣ Link Signal Splitters ◈ Electrical –Ethernet tap, DS-3 tap, etc. ◈ Optical –ordinary optical splitter –independent of physical and data-link layer protocols ▣ High Performance Packet Capture Cards ◈ Model A: for lower speed links –Ethernet, FastEthernet, DS-3/(E3) ◈ Model B: for middle speed links –ATM at OC-3, and POS at OC-3, OC-12 (622Mbps)

27. -26- Wise* TrafView User Interface ▣ Web-based Interface ◈ simple ◈ easy to use ◈ intuitive ◈ portable ▣ A web site for each measurement site can be easily established ◈ Autonomous authentication and authorization can be supported

28. -27- Wise* TrafView GUI (Traffic Report)

29. -28- Wise* TrafView

30. -29- Wise* TrafView GUI (Traffic Matrix)

31. -30- Wise* TrafView 5. SUMMARY

32. -31- Wise* TrafView The Merits of Wise* TrafView ▣ Transparent Packet Capture ◈ Complete independence of the existing networking equipment ▣ Flow-based Measurement and Analysis ◈ Reduced load ◈ Higher degree of recognition ▣ Understanding Application Specific Contexts ◈ By means of enhanced application recognition algorithms, sub-flows can be detected ▣ Scalable ◈ Can scale up from tens of Mbps to Gbps ◈ Supports various physical and data-link layer technologies ▣ Highly Extensible and Adaptable ◈ Easy configuration with ARCL

33. -32- Wise* TrafView Thank you! Q&A Contact: jungsp@etri.re.kr, chunghs@etri.re.kr, choits@etri.re.kr, tsjeong@etri.re.kr