60 Days of Basic Naughtiness Probes and Attacks Endured by an Active Web Site 16 March 2001 This is part of a larger project – to analyze and track attack and probe methods and sources. A holistic view of site probes and attacks. To create an early warning and verification/validation system/site for others to use. To track particularly popular source netblocks and assist the netblock owners with proper defense and mitigation. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

60 Days of Basic Naughtiness Statistical analysis of log and IDS files. Statistical analysis of a two-day DDoS attack. Methods of mitigation. Questions. The analysis was performed using copious amounts of Perl, sed, awk, and coffee.  The methods are still being honed – they are far from perfect. Started with a series of simple questions raised by a manager : “When did this attack start?” “Did it slowly gain speed? Should we have seen it coming?” “When will it end? Can we predict the end of the attack?” Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Rob Thomas robt@cymru.com About the Site Production site for several (> 4) years. Largely static content. No e-commerce. Layers of defense – more on that later! Agreed to keep the site name and related data confidential. I am always looking for more log files! Please feel free to share anything you have. Send it to my e-mail address. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Rob Thomas robt@cymru.com About the Data Data from router logs. Data from IDS logs. Snapshot taken from 60 days of combined data. Data processed by several home-brew tools (mostly Perl and awk). The router and IDS logs are not in sync, unfortunately. This is the first significant effort with copious amounts (942MB) of log data. Previous efforts utilized a significantly smaller sample size. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Definition of “Naughty” Any traffic that is logged by a specific “deny” ACL. Any traffic that presents a pattern detected by the IDS software. The two log sources are not necessarily synchronized. ACLs – deny and log RPC, bogon source addresses, etc. IDS – detect SYN floods, Backorifice probes, etc. The data files do not overlap perfectly. This would be ideal for a completely holistic view of site probes and attacks. Remember – NTP is your friend! Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Daily Probes and Attacks TCP and UDP Probes and Attacks – ICMP not counted. Average – 529.00 Standard deviation – 644.10! 60 Day Low – 83.00 60 Day High – 4355.00 The standard deviation is higher than the mean! In other words, the ebb and flow of probes and attacks is of a greatly variable nature, and therefore difficult to model. Note the 60 day low figure of 83.00. This means that there is never a completely “safe” day! Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Daily Probes and Attacks Note that never a day goes by without at least 83 probes and/or attacks! The miscreants hit the site 24 x 7 x 365. Why so many? Because the miscreants do NOT share data! Elucidate this point (IRC wars). Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Weekly Probes and Attacks There is no steady-state. Attacks come in waves, generally on the heels of a new exploit and scan. Certain types of scans (e.g. Netbios) tend to run 24x7x365. Proactive monitoring, based on underground and public alerts, will result in significant data capture. There is no “steady-state,” and there is no “normal” week or day. The vulnerability du jour (e.g. BIND hole, sendmail hole) tends to wane over a two to three month period. When an alert comes out – particularly one that lists a port or ports – fire up an ACL and watch the log entries grow.  Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Weekly Probes and Attacks Trend Analysis Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Hourly Probes and Attacks Myth: “Most attacks occur at night.” An attacker’s evening may be a victim’s day – the nature of a global network. Truth: Don’t plan based on the clock. We are now part of a global network – a network that NEVER sleeps. The average DDoS attack lasts between 24 and 72 hours – the zombies don’t need a break. Take a guess – what hour of the day will be the most utilized by the miscreants to hit this site? Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Hourly Probes and Attacks Trend Analysis Rob Thomas robt@cymru.com http://www.cymru.com/~robt

UDP Probes and Attacks Top Five Destination Ports First – 137 NETBIOS Second – 53 DNS Third – 27960 Fourth – 500 ISAKMP Fifth – 33480 (likely UNIX traceroute) Rob Thomas robt@cymru.com http://www.cymru.com/~robt

UDP Probes and Attacks Trend Analysis Rob Thomas robt@cymru.com http://www.cymru.com/~robt

TCP Probes and Attacks Top Five Destination Ports First – 3663 (DDoS Attack) Second – 0 Reserved (DDoS Attack) Third – 6667 IRC (DDoS Attack) Fourth – 81 (DDoS Attack) Fifth – 21 FTP-control With the exception of TCP 21, all other listed ports were part of a DDoS attack attempt. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

TCP Probes and Attacks Trend Analysis TCP port zero remains very popular with those who target this site. TCP port 21 (FTP) probes have been supplanted by other probes, such as DNS. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Source Address of Probes and Attacks Note that Multicast and experimental source address packets account for 53% of the naughty packets. Thus 53% of the nasty packets are blocked at our border! What about the other classes? Are the sources therein all legitimate? Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Source Address of Probes and Attacks Class A bogon source percentage – 48.08%! Class B bogon source percentage – 20.80%. Class C bogon source percentage – 11.87%. Class D and E bogon source percentage – 100.00%. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Source Address of Probes and Attacks Bogon source attacks still common. Of all source addresses, 53.39% were in the Class D and Class E space. Percentage of bogons, all classes – 66.85%! This is good news – prefix-list, ACL defense, and uRPF will block 66.85% of these nasties! Block bogons, both inbound and outbound, at your border! Add anti-spoofing at your egress points so that only source addresses within your allocated netblocks may pass. Three attack types: Spoof the source using bogon addresses. Spoof the source using legitimate addresses. No spoofing. If all networks performed these basic steps, how successful would spoof attacks be? Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Source Region of the Naughty A dangerously misleading slide The location of the miscreant is difficult to determine because of: Spoof attacks. Zombies. Legacy netblock assignments. Key point – the RIR or netblock does not necessarily indicate the location of the miscreant(s)! Statistics can be misleading. Proper analysis requires a certain degree of “clue.” Latest groups are from Brasil and Romania. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Intrusion (attempt) Detection IDS is not foolproof! Incorrect fingerprinting does occur. You can not identify that which you can not see. IDS is really a misnomer – it should be Intrusion Attempt Detection System, or IADS. Intrusion Detection is easy – you will know your hull has been breached when your web page reads “Hackers love Ramen!”  IDS can be overwhelmed. Stick is nothing new – we have likely all seen it before. Keep your IDS database current! Place your IDS tool where it will provide the best detection and analysis. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Top Five IDS Detected Probes NetBus – TCP 12345 Backorifice – UDP 31337 TFTP – UDP 70 IDENT – TCP 113 Deep Throat – TCP/UDP 2140 Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Top Five Detected IDS Probes Notice how the probes converge around day 13 and particularly days 26 and 27. This is not likely to be a coincidence! Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Top Five IDS Detected Attacks Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Top Five IDS Detected Sources Azerbaijan – ISP USA 01 – Consulting Company South Korea – ISP USA 02 – ISP dialup pool Canada – Cable modem Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Top Five IDS Detected Sources Notice how netblock B (USA 01) trends sharply upward around days 24 through 25. In the raw data spreadsheet, the sudden appearance of netblock B coincides with the sharp increase in probe activity as seen in the Top Five Probes slide. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Match a Source with a Scan Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Rob Thomas robt@cymru.com Two Days of DDoS Attack that resulted in 10295 hits on day one and 77466 hits on day two. Attack lasted 25 hours, 25 minutes, and 44 seconds. Quasi-random UDP high ports (source and destination), small packets. This was a relatively mild attack, and this makes for simple analysis. However, the analysis steps are the same regardless of attack intensity. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Rob Thomas robt@cymru.com Two Days of DDoS Perhaps as many as 2000 hosts used by the attackers. 23 unique organizations. 9 different nations located in the Americas, Europe, and Asia. Source netblocks all legitimate. The use of legitimate source addresses makes defense that much more difficult. This was an extremely mild DDoS – perhaps a test of things to come. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Rob Thomas robt@cymru.com Two Days of DDoS As with all DDoS attacks, the ramp up time was quite fast – on the order of several seconds. The end came just as suddenly. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Rob Thomas robt@cymru.com Two Days of DDoS Note the rise and fall of myriad sources. This is not uncommon during even the most intense DDoS attacks, and can be due to several factors, such as: Congestion at the attacker site(s) Reactionary filtering (local and intermediate) during the attacks Zombie control issues Insert the IRC “out of control zombie story” here. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Site Defense and Attack Mitigation While you can not prevent an attack, you can choose how to react to an attack. Layers of defense that use multiple tools. Layers of monitoring and alert mechanisms. Know how to respond before the attack begins. Each layer should protect the layer beyond. Each layer should integrate with the layers on either side. Each layer should be untrusted by the next site-facing layer. Funnel the traffic, filtering at an ever more granular level of detail. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Site Defense and Attack Mitigation Border router Protocol shaping and filtering. Anti-bogon and anti-spoofing defense (uRPF), ingress and egress filtering. NetFlow. IDS device(s) Attack and probe signatures. Alerts. Block bogons and prevent spoofing! The presence of an IDS device does not obviate the need to have the other filtering devices logging (ACL logs, rule base logs, NetFlow, etc.). Compare log entries often – remember, NTP is your friend. Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Site Defense and Attack Mitigation Border firewall Port filtering. Logging. Some IDS capability. End systems Tuned kernel. TCP wrappers, disable services, etc. Crunchy through and through! Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Site Defense and Attack Mitigation Don’t panic! Collect data! The good news - you can survive! Rob Thomas robt@cymru.com http://www.cymru.com/~robt

References and shameless self advertisements  RFC 2267 - http://rfc.net/rfc2267.html Secure IOS Template – http://www.cymru.com/~robt/Docs/Articles/secure-ios-template.html Secure BGP Template – http://www.cymru.com/~robt/Docs/Articles/secure-bgp-template.html UNIX IP Stack Tuning Guide – http://www.cymru.com/~robt/Docs/Articles/ip-stack-tuning.html Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Rob Thomas robt@cymru.com Any questions? Rob Thomas robt@cymru.com http://www.cymru.com/~robt

Rob Thomas robt@cymru.com Thank you for your time! Thanks to Jan, Luuk, and Jacques for inviting me to speak with you today. Thanks to Surfnet/CERT-NL for picking up the travel. Thanks for all of the coffee!  Rob Thomas robt@cymru.com http://www.cymru.com/~robt