-1- Wise* TrafView Wise * TrafView ETRI’s Content-aware Internet Application Traffic Measurement and Analysis System APAN Network Technology WS January 29, 2004 IP Networking Technology Team, ETRI {jungsp, chunghs, choits,

-2- Wise* TrafView Contents ▣ Current Internet Application Traffic Characteristics ▣ Wise* TrafView : Our Approach ▣ Wise* TrafView : Implementation and Deployment Experiences ▣ Summary

-3- Wise* TrafView Measurement Application Areas ▣ Network Problem Determination and Analysis ▣ Traffic Report Generation ▣ Intrusion & Hacking Attack (e.g., DoS, DDoS) Detection ▣ Service Level Monitoring (SLM) ▣ Network Planning ▣ Usage-based Billing/Accounting (both between SPs and SP- and-Customer) ▣ Customer Relationship Management (CRM) ▣ Marketing

-4- Wise* TrafView 2. CURRENT INTERNET APPLICATION TRAFFIC CHARACTERISTICS

-5- Wise* TrafView Current Internet Traffic Characteristics ▣ High-speed networks (Mbps  Gbps  Tbps) ▣ High-volume traffic ▣ Variety of Applications ◈ Streaming media (Windows Media, Real Media, Quicktime) ◈ P2P traffic ◈ Network Games ◈ Network Security Attacks ◈ Etc.

-6- Wise* TrafView Application Recognition(1) ▣ Limitations of port-based recognition ◈ The port database maintained by IANA doesn’t reflect the real-world situation –Most newer applications simply do not register their ports –Sometimes they even invade well-known port area to pass thorough firewalls ◈ Most bandwidth hogs, nowadays, dynamically allocate ports –They are not linked up with any fixed ports!

-7- Wise* TrafView Port/Application Port-based Accounting Contents-aware Accounting 80/HTTP67 GB 59.1 GB (11.8% reduced) 21/FTP_CTRL0.29 GB0.28 GB 20/FTP_DATA43 GB42 GB ?/FTP_DATA_PASSIVEn/a 6 GB (14.3% of FTP_DATA, 2% of the total volume) 5003/?692 MB HTTP: 13.2 MB BUGS_MUSIC: MB EDONKEY: MB etc.: 85.7 MB PosTech Traffic Breakdown - PosTech Campus Network (24h sum in May, 304GB total volume) Application Recognition(2) : Trend in Internet Application Traffic Characteristics

-8- Wise* TrafView Application Recognition(3) ▣ Many applications require to be identified by payload inspection ▣ Why is payload inspection necessary? ◈ Several applications can use the same port number ◈ Identification error can be occurred by ephemeral port number ◈ Some applications can use a dynamic port number ◈ Etc.

-9- Wise* TrafView ▣ Application example : Passive FTP Application Recognition(4) client.1302 server.21 (FTP_CTRL_REQ) server.21 (FTP_CTRL_REP) client.1302 client.1303 server.20 (FTP_DATA_DOWN) server.20 (FTP_DATA_UP) server (FTP_DATA_PSV_UP) client.1306 server (FTP_DATA_PSV_DOWN) % ls % passive % get wmggw.mp3 % quit % ftp server 49152

-10- Wise* TrafView Why Port-based Approach is not enough? ▣ Non-flow based measurement ◈ Not enough for the above requirements ▣ Typical Flow-based Measurement (like NetFlow TM, cflowd, LFAP) ◈ Typically a flow is defined as a set of packets passing an observation point in the network during a certain time interval and having a set of common properties ◈ 5-tuple packet header fields are used for this ◈ New applications such as P2P, streaming and network games have characteristics of dynamic port allocation ▣ More Detailed Analysis is needed!! ◈ Typical Flow-based Measurement is not enough ◈ Need more detailed analysis depending on applications –It may require content filtering

-11- Wise* TrafView 3. Wise* TrafView : OUR APPROACH

-12- Wise* TrafView Motivation ▣ Develop precise Internet application traffic measurement and analysis system ◈ Precise application analysis ◈ Passive flow-based measurement ◈ Sub-transaction(flow) level detailed application analysis ◈ Pseudo-realtime analysis ◈ No loss capture and analysis ◈ No sampling but capturing all ◈ For various Internet measurement purposes

-13- Wise* TrafView Flow Concept ▣ A “flow” is ◈ a sequence of packets whose are all identical ▣ Why flow? ◈ The size of entire raw packet streams for a given unit time are prohibitively enormous to be analyzed in time ◈ Each individual packets in a flow contain duplicate information ◈ Packets in the same flow are correlated; we can identify more packets which were previously categorized as unknown application a packet a distinctive signature of application “X” a flow generated by application “X” Now, these pkts can also be identified as “X”

-14- Wise* TrafView Internet Application Classification ▣ Type S: Simple Application Type ◈ for an application which uses a well-known port number or which uses a registered port number but is popularly used ◈ Applications : WWW, FTP, SMTP, BGP, etc. ▣ Type P: Payload Application Type ◈ for an application which uses a registered port number but requires payload inspections for precise classification ◈ Applications : HTTP_ALT(8080,8081,9000), MSNMessenger( ), KAZZA(1214), … ▣ Type R: Reverse Application Type ◈ for an application which uses a registered but requires comparison with a correlated reverse flow for the precise classification ◈ Applications : eDonkey down, WINMX down, GuruGuru BBS(9999)… ▣ Type C: Co-related Application Type ◈ for an application which uses a dynamic port number assignment ◈ Applications : Passive FTP, RTSP, Windows Streaming, …

-15- Wise* TrafView Capture Agent Analysis Server Database GUI... NIC IPCAP Card... NIC IPCAP Card splitter flow and packet records (NFS) recognition and analysis results (ODBC) ARCL Config-File... System Architecture Overview

-16- Wise* TrafView Agent : Generating Flow & Packet Records ▣ Carries on simple filtering and signature matching functions ▣ Generates flow records & packet records ◈ Flow record –For flow information –Fields : IP addr, port, protocol, flow duration, packets, bytes, … ◈ Packet record –for individual packet –Fields : timestamp, TOS, TTL, TCP flags, payload, … –Important for analysis server’s precise application identification ◈ This procedure aggregates and organizes the traffic information and reduces the amount of traffic volume transferred to the server

-17- Wise* TrafView Analysis Server : Enhanced Application Recognition ▣ Wise* TrafView utilizes some enhanced proprietary recognition mechanisms in a comprehensive way ◈ Application specific signature matching, ◈ temporal and spatial flow correlation, ◈ dynamic port recognition and utilization, and ◈ some heuristics ▣ Not only capable of discriminating applications, but also their sub-flows ◈ e.g., HTTP  HTTP_REQ, HTTP_REP, HTTP_REQACK, etc.

-18- Wise* TrafView Analysis Server : AS and Country Mapping ▣ Identifying flow sources and destinations ◈ Both source and destination IP address of a flow are mapped to ASes and finally to country codes ◈ This helps to locate the source and the sink of a flow –enables discrimination among transit, inbound, and outbound traffic flows

-19- Wise* TrafView Application Recognition Configuration Language (ARCL) ▣ Configurability and Adaptability ▣ Why adaptability so important? ◈ The highly frequent nature of Internet applications’ appearance and disappearance ◈ Swift mutation of applications ◈ Localization of the use patterns of applications ▣ Wise* TrafView copes with the problem by introducing ARCL (Application Recognition Configuration Language) ▣ By taking advantage of ARCL, Wise* TrafView ◈ doesn’t need to be re-built or re-installed by any module for extending or modifying recognition coverage; editing the configuration in ARCL and re-enforcing suffices

-20- Wise* TrafView Config-file by ARCL application WWW { port_rep_name HTTP port 80 protocol TCP{ // S type decision_group HTTP_REQ_REP_ACK { src_port >= 1024 dst_port == 80 } decision_group HTTP_REP_REQ_ACK { src_port == 80 dst_port >= 1024 } port_rep_name HTTP_ALT port 8080 protocol TCP{ // P type src_disc_pattern=="HTTP" in pkt 0-2 at byte ( dst_disc_pattern=="GET" in pkt 0-3 at byte || dst_disc_pattern=="POST" in pkt 0-3 at byte ) decision_group HTTP_ALT_REQ_REP_ACK { src_port >= 1024 dst_port == 8080 } decision_group HTTP_ALT_REP_REQ_ACK { src_port == 8080 dst_port >= 1024 } application EDONKEY { // R type port_rep_name EDONKEY_DOWN port 4662 protocol TCP{ dst_disc_pattern=="0xe33d000000" in pkt 2-3 at byte decision_group EDONKEY_DOWN_REQ_REP_ACK { src_port >= 1024 dst_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 } decision_group EDONKEY_DOWN_REP_REQ_ACK { src_port == 4662 ~ 4666 || 4242 || 4224 || 4660 || 5555 dst_port >= 1024 } …… } application FTP { // C type port_rep_name FTP port 21 protocol TCP{ src_ref_pattern=="r/227 Entering Passive Mode \(\d{1,3},\d{1,3},\d{1,3},\d{1,3},(\d{1,4}),(\d{1,4})\)/$src_port = atoi($1)* atoi($2)" in pkt any at byte 0-35 induce FTP_DOWN_P decision_group FTP_REQ_REP_ACK { src_port >= 1024 dst_port == 21 } decision_group FTP_REP_REQ_ACK { src_port == 21 dst_port >= 1024 }

-21- Wise* TrafView 4. Wise* TrafView : IMPLEMENTATION & DEPLOYMENT EXPERIENCES

-22- Wise* TrafView Deployment Experiences ▣ ETRINet ◈ Link speed : 100Mbps FastEthernet, using libpcap ◈ Traffic volume : 70Mbps ◈ Period : May 2003 – Current ◈ Analysis result : S(52.83%), P(9.99%), R(2.38%), C(4.92%), Unknown(28.88%) ▣ Postech ◈ Link speed : 1Gbps Ethernet, using libpcap ◈ Traffic Volume : 60 – 70Mbps ◈ Period : May 2003(1week) ▣ Univ. of Andong ◈ Link speed : FastEthernet, using capturing card developed by ETRI ◈ Traffic volume : 60-70Mbps ◈ Period : Oct Current ▣ Other experiences ◈ Deployment on the International link of one of Korean Internet Exchange point using OC-3 POS card developed by ETRI

-23- Wise* TrafView Port/Application Port-based Accounting (A) Contents-aware Accounting (B) Accuracy (A/B) 80/HTTP67 GB 59.1 GB (11.8% reduced) 0.882/1.0 21/FTP_CTRL0.29 GB0.28 GB0.965/1.0 20/FTP_DATA43 GB42 GB0.977/1.0 ?/FTP_DATA_PAS SIVE n/a 6 GB (14.3% of FTP_DATA, 2% of the total volume) 0.0/ /?692 MB HTTP: 13.2 MB0.0/1.0 BUGS_MUSIC: MB0.0/1.0 EDONKEY: MB0.0/1.0 etc.: 85.7 MB0.0/1.0 - PosTech Campus Network (24h sum in May, 304GB total volume) PosTech Traffic Analysis Result

-24- Wise* TrafView ▣ Hardware ◈ For lower speed links (<= 622Mbps) –Capture agent –high performance PC: Zeon 2.4GHz * 2 + CPU, 2GB+ RAM –Analysis server –high performance PC: Zeon 2.8GHz * 2 + CPU, 1GB+ RAM, 100GB+ HDD ◈ For Higher speed links ( > 1 Gbps, under developing) –Clustered capture system –Hardwired logic for supporting wire-speed processing ▣ Software ◈ Capture agent –Linux ◈ Analysis server –Linux, MySQL System Spec.(1)

-25- Wise* TrafView System Spec.(2) ▣ Link Signal Splitters ◈ Electrical –Ethernet tap, DS-3 tap, etc. ◈ Optical –ordinary optical splitter –independent of physical and data-link layer protocols ▣ High Performance Packet Capture Cards ◈ Model A: for lower speed links –Ethernet, FastEthernet, DS-3/(E3) ◈ Model B: for middle speed links –ATM at OC-3, and POS at OC-3, OC-12 (622Mbps)

-26- Wise* TrafView User Interface ▣ Web-based Interface ◈ simple ◈ easy to use ◈ intuitive ◈ portable ▣ A web site for each measurement site can be easily established ◈ Autonomous authentication and authorization can be supported

-27- Wise* TrafView GUI (Traffic Report)

-28- Wise* TrafView

-29- Wise* TrafView GUI (Traffic Matrix)

-30- Wise* TrafView 5. SUMMARY

-31- Wise* TrafView The Merits of Wise* TrafView ▣ Transparent Packet Capture ◈ Complete independence of the existing networking equipment ▣ Flow-based Measurement and Analysis ◈ Reduced load ◈ Higher degree of recognition ▣ Understanding Application Specific Contexts ◈ By means of enhanced application recognition algorithms, sub-flows can be detected ▣ Scalable ◈ Can scale up from tens of Mbps to Gbps ◈ Supports various physical and data-link layer technologies ▣ Highly Extensible and Adaptable ◈ Easy configuration with ARCL

-32- Wise* TrafView Thank you! Q&A Contact: