High-Performance Network Anomaly/Intrusion Detection & Mitigation System (HPNAIDM) Yan Chen Department of Electrical Engineering and Computer Science Northwestern University Lab for Internet & Security Technology (LIST)

Current Intrusion Detection Systems Mostly not scalable to high-speed networks –Slammer worm infected 75K machines in <10mins –Host-based schemes inefficient & user dependent Statistical detection unscalable for flow- level detection Mostly simple signature-based –Cannot detect unknown and polymorphic attacks Cannot differentiate malicious events with unintentional anomalies

High-Performance Network Anomaly/Intrusion Detection and Mitigation System (HPNAIDM) Online traffic recording [SIGCOMM IMC 2004, IEEE INFOCOM 2006, ToN to appear] –Reversible sketch for data streaming computation –Record millions of flows (GB traffic) in a few hundred KB –Infer the key (eg, src IP) even when not directly recorded Online sketch-based flow-level anomaly detection [IEEE ICDCS 2006] [IEEE CG&A, Security Visualization 06] –As a first step, detect TCP SYN flooding, horizontal and vertical scans even when mixed

HPNAIDM (II) Integrated approach for false positive reduction Polymorphic worm detection (Hamsa) [IEEE Symposium on Security and Privacy 2006] Accurate network diagnostics [ACM SIGCOMM 2006] Scalable and robust distributed intrusion alert fusion with DHT [ACM SIGCOMM Workshop on Large Scale Attack Defense 2006]

HPNAIDM Architecture Remote aggregated sketch records Streaming packet data Part II Per-flow monitoring & detection Reversible sketch monitoring Filtering Sketch based statistical anomaly detection (SSAD) Local sketch records Sent out for aggregation Per-flow monitoring Normal flows Suspicious flows Intrusion or anomaly alarms Keys of suspicious flows Keys of normal flows Data path Control path Modules on the critical path Signature -based detection Polymorphic worm detection (Hamsa) Part I Sketch- based monitoring & detection Modules on the non-critical path Network fault diagnosis

IRC-based Botnet Detection on Routers

Trend on Botnets Total infected bot hosts 800, ,000 [CERT CA ] Symantec identified an average of about 10,000 bot infected computers per day [Mar Internet Security Threat Report] # of Botnets - increasing Bots per Botnet - decreasing –Used to be 80k-140k, now 1000s More firepower: –Broadband (1Mbps Up) x 100s = OC3

Geographical Distribution of Bots Note that this doesn’t reflect where the attackers are.

Trend on Botnets II Distribution of Command and Control servers –Top 3: USA (48%), South Korea (9%) and Canada(6%) US also experienced the highest percentage of growth in bot-infected computers –The number of bot-infected computers increased by 39% in the second half of 2005 –Wide adoption of broadband ? Bot-related malicious code reported to Symantec accounted for 20% of the top 50 malicious code reports, up from 14%.

Problem Definition For an ISP/enterprise network operator monitoring at the edge router/gateway, how to detect botnet server/channel even when such traffic is encrypted ? Identify attacker Disable botnets Internet Edge network botnet server/channel ?

Existing Work on Botnet Detection Mostly honeypot based approaches –Trap bots and analyze their behavior –Eg, Honeynet project, U Michigan [SRUTI 05] –Hard to generate traffic signatures for network detection Identify botnet channel –Assuming to know the IRC traffic first, look for channel w/ majority of hosts performing TCP SYN scans [SRUTI 06] –Hard to differentiate from P2P & game traffic –Bots w/ emerging infection scheme (SMTP) ?

Existing Work on Botnet Detection II IDS-based approach like Snort –Use port numbers and key words (e.g., PRIVMSG, lsass, NICK, etc.) –High false positive and/or false negative –E.g., what about encrypted bot channel ? –Complementary to our approach

Our Approach Two steps: Separate IRC traffic from normal traffic Identify botnet traffic in the IRC traffic

Separating IRC Traffic from Other Traffic Key characteristic: relay (broadcast) –Upon an incoming packet of size x, broadcast a packet to one or many different IPs (with packet size similar to x) Packet size: median packet size < 100B Duration: average life time 3.5 hours Port numbers: 6667, 6668, 6669, 7000, 7514 –But IRC/botnets can also run on non-standard ports Combine all these

Preliminary Analysis IRC traffic observed at an university edge router –Mostly packet headers with limited payload –Collected in April, 2006 Data size:378M Duration:5 days # of packets Mean packet length (bytes) 164 # of sessions664 Mean session duration (s) # of IRC servers54 # of IRC clients39 PING39638 PONG39672 PRIVMSG JOIN41905 QUIT34439 ISON15129 WHO9144 MODE6804

CDF of Session Durations

Packet Length Distribution Large packets caused by membership listing

These Metrics Are Not Enough ! Online games, and P2P systems Relay broadcast: –Game update, query broadcast from supernodes, e.g., Gnutella (not for all P2P systems) Small average packet size –FPS (first person shooting), e.g., CounterStrike »All UDP packets and w/ packet size dist 40 ~ 120B –RTS (real time strategy), e.g., Warcraft III »TCP packets, and the packet size is extremely small, 5~10B payload –Supernodes of P2P only broadcast small query packets w/o real file transfer Long session durations

Additional Characteristics for IRC Traffic IRC traffic usually generated through human typing or bot command execution report –Small packet frequency and throughput per IP –Key differentiator from the RTS games »Each client sends out at least 5~10 packets per second Still, what about P2P? –Existing traffic study do not have the answer »Transport layer identification of P2P traffic [IMC 04] use port # to separate IRC traffic »Study of Internet chat systems [IMC03] use port # and keywords to identify IRC traffic –Our approach: complement w/ active probing

Identify Botnet Traffic (with Packet Header only) When attacker sends command to bots, they will mostly finish within certain period and send back similar replies –Identify groups of IPs that belong to different channels –Identify bot channel which has a large number of non-control messages of similar sizes at the same time Bot repeatedly connect to IRC server when they fail the connection –Even ignore error messages from IRC servers, e.g., connecting too fast or nickname used

Identify Botnet Traffic (with payload) Most normal IRC server un-encrypted Look for commands of keywords –Eg, bot*, ddos*, scan* in Agobot Check content similarity of client replies –Most bots’ replies are similar, e.g., using Hamming distance

Preliminary Analysis Packet traffic from a botnet IRC server at a compromised machine Data size:6MB Duration:7 minutes on Jun 05 Number of packets 65,135 Mean packet length (bytes) 104 # of bots1221 Total21,004 PING6596 PONG5846 PRIVMSG971 JOIN1198 QUIT52 NOTICE3373

Content Analysis [:IRC] PRIVMSG #r00t# :[nickname]: lsass: exploited ( ) 210, 201 [:IRC] PRIVMSG #r00t# :[nickname]: ftp: on , 149 [:IRC] PRIVMSG #scan :[nickname] :CSendFile(0x0546EFA0h): Transfer to finished. 105, 97 [nickname] PRIVMSG #bz-sniff :FTP sniff " :4868" to " :8500": - "USER administrator " 60, 45 The message length of each type are very similar, because they only change IP, port number or number of bytes

Content Analysis II PRIVMSG #sniff :HTTP sniff " :80" to " :1583": - "HTTP/ OK Server: Microsoft-IIS/5.0 Date: Wed, 13 Oct :33:19 GMT Content-Length: 46 Content-Type: text/html Set- Cookie: BID= ; expires=Mon, 12-Oct :00:00 GMT; path=/ Set-Cookie: PID=1156; expires=Mon, 12-Oct :00:00 GMT; path=/ Cache-control: private 5.00_5.00_UG " 17 PRIVMSG #vuln :VULN sniff " :80" to " :4690": - "HTTP/ OK Server: Apache/ (Unix) (Red-Hat/Linux) mod_ssl/ OpenSSL/0.9.6b DAV/1.0.3 mod_perl/1.26 mod_oas/5.6.1 mod_cap/2.0 P3P: CP="NON NID PSAa PSDa OUR IND UNI COM NAV STA",policyref="/w3c/p3p.xml" P3P: CP="NON NID PSAa PSDa OUR IND UNI COM NAV STA",policyref="/w3c/p3p.xml" Last-Modified: Mon, 04 Oct :22:54 GMT ETag: "10003e-2a cd3e" Accept-Ranges: bytes Content-Length: Content-Type: image/gif Date: Wed, 13 Oct :36:38 GMT Connection: keep-alive GIF89a 24 Note that Sniff report can vary a lot in length and content

Bots Making Repeated Connection Attempts Even after receiving error messages, e.g., connecting too fast or nickname used

Summary Goal: Detect botnet server/channel at edge network routers/gateways even when such traffic is encrypted Separate IRC traffic from normal traffic Identify botnet traffic in the IRC traffic Contact: Yan Chen