1. Information Networking Security and Assurance Lab National Chung Cheng University Intrusion Detection Testing and Benchmarking Methodologies Nicholas Athanasiades, Randal Abler, John Levine, Henry Owen, and George Riley School of Electrical and Computer Engineering Georgia Institute of Technology

2. Information Networking Security and Assurance Lab National Chung Cheng University 2 1. Introduction Beginning of the Intrusion Detection Evaluation  DARPA(1998~1999)  LARIAT (Lincoln Adaptable Real-time Information Assurance Test-bed)(2000~2001) Most common methodologies Traffic generation is one of the most difficult ones  Synthetic traffic not represent the realities of an actual network SmartBits Scripting tools

3. Information Networking Security and Assurance Lab National Chung Cheng University 3 2. Existing Tools and Testing Methodologies A. DARAPA Environment B. LARIAT Environment C. Nidsbench and IDS Wakeup D. IDSwakeup E. Flame Thrower F. WebAvalanche/WebReflector G. Tcpreplay H. Fragrouter I. Hping2 J. Iperf

4. Information Networking Security and Assurance Lab National Chung Cheng University 4 2. Existing Tools and Testing Methodologies A. DARAPA Environment  Approach An off-line (Tune and optimize) and an on-line (actual testing) evaluation executed Tcpreplay  Protocol/traffic activity HTTP, X window, SQL, SMTP, DNS, FTP, POP3, Finger, Telnet, IRC, SNMP, and Time

5. Information Networking Security and Assurance Lab National Chung Cheng University 5 2. Existing Tools and Testing Methodologies A. DARAPA Environment SolarisSunOSLinux Denial of Service (11 types, 43 instances) Back, Neptune, Ping of death, Smurf, syslog, Land, apache2, Mailbomb, Process table, UDP storm Back, Neptune, Ping of death, Smurf, Land, apache2, Mailbomb, Process table, UDP storm Back, Neptune, Ping of death, Smurf, teardrop, Land, apache2, Mailbomb, Process table, UDP storm Remote to Local (14 types, 17 instances) Dictionary, ftp-write, guest, phf, http tunnel, xlock, xsnoop Dictionary, ftp-write, guest, imap, phf, named, http tunnel, sendmail, xlock, xsnoop User to Root (7 type, 38 instances) Eject, ffbconfig, Fdformat, ps Loadmodule, psPerl, xterm Surveillance/ Probe (6 types, 22 instances) Eject, nmap, Port sweep, Satan, mscan, saint Figure 1 Attacks in the 1998 DARPA evaluation

6. Information Networking Security and Assurance Lab National Chung Cheng University 6 2. Existing Tools and Testing Methodologies A. DARAPA Environment  1999: the goals shifted to testing complete systems  Changes and additions Victim Windows NT added New stealthy attacks added Two new types of analysis performed An analysis of misses and high-scoring false alarms Participants were allowed to submit information aiding in the identification of many attacks and their appropriate response Detection of novel attacks without first training

7. Information Networking Security and Assurance Lab National Chung Cheng University 7 2. Existing Tools and Testing Methodologies B. LARIAT Environment  LARIAT “emulates the network traffic from a small organization connected to the Internet”  Many phases Network discovery phase Then, initializes the network and configures the hosts The test’s conditions are set up  Traffic generation is done through the use of defined service models Modified a Linux Kernel that allow their software to generate background traffic  Part of a government project and not publicly available

8. Information Networking Security and Assurance Lab National Chung Cheng University 8 2. Existing Tools and Testing Methodologies C. Nidsbench  A NIDS Test Suite released in 1999  Made up of the components tcpreplay, idtest and fragrouter D. IDSwakeup  Like Nidsbench  It generates false attacks, a false positive test utility  Consists of IDSwakeup and utilizes hping and iwu E. Flame Thrower  Commercial load stress tool used to identify network infrastructure weaknesses  Produces transaction in order to test network infrastructure and applications  Supports HTTP/HTTPS 1.0, 1.1 and SSL  It can emulate over two million IP address  FirewallStressor measure throughput under attack conditions  Flame Thrower intended for testing firewalls

9. Information Networking Security and Assurance Lab National Chung Cheng University 9 2. Existing Tools and Testing Methodologies F. WebAvalanche/WebReflector  Commercial network appliances used in the testing of IDS  WebAvalanche is a stress-testing appliance  WebReflector emulates the behavior of large Web, application and data server environments  Support such as HTTP 1.0/1.1, SSL, RTSP/RTP and FTP  Measure percent dropped packets, latencies, maximum number of users and new user arrival rates G. Tcpreplay  Allows captured traffic to be played back on a network at different speeds  Tcpdump or snoop

10. Information Networking Security and Assurance Lab National Chung Cheng University 10 2. Existing Tools and Testing Methodologies H. Fragrouter  An attack generation tool  For testing anti-evasion techniques and fragmentation queues I. Hping2  A command-line packet assembler and analyzer  Allows one to create and transmit custom ICMP, UDP, and TCP packets  Fingerprint remote operating systems J. Iperf  Measures bandwidth, delay jitter and datagram loss  Used as a background traffic source

11. Information Networking Security and Assurance Lab National Chung Cheng University 11 4. Examples of Intrusion Detection Evaluation Environments DARPA Like Environment Custom Software Advanced Security Audit Trail Analysis on Unix Vendor Independent Testing Lab Trade Magazine Evaluation

12. Information Networking Security and Assurance Lab National Chung Cheng University 12 DARPA Like Environment 5 components  Traffic generating  Victim was “an anonymous FTP server running on a Sun UltraSparc-1 using a Solaris 2.5 OS  Attack Injection programs  The in house reference programs counted the number of hung connection at the victim server as a measure of attack effectiveness. They used a metric called virulence. Virulence described the intensity of an attack situation.  The evaluation method was to use 10, 15, 30, 40 and 60 attacking hosts each utilizing rates of varying rates of attacks per second.

13. Information Networking Security and Assurance Lab National Chung Cheng University 13 Custom Software A software platform that simulates intrusions and tests IDS effectiveness Criteria used included  Broad Detection Range  Economy in resource usage  Resilience to stress The benchmark platform was base on Expect and Tool Command Language Distributed Programming (TCL-DP) package

14. Information Networking Security and Assurance Lab National Chung Cheng University 14 A dvanced S ecurity audit trail A nalysis on uni X The test consisted of the following scenarios  Trojan horse  Attempted break-ins  Masquerading  Suspicious connections  Black listed addresses  Nosing: numerous moves through directories  Privilege abuse

15. Information Networking Security and Assurance Lab National Chung Cheng University 15 Vendor Independent Testing Lab NSS tests a broad range of features of IDS  Convenience: ease of installation, deployment and management  UI: reporting and alerts delivered  Attack signatures  Accuracy  Peripheral issues like licensing, documentation and log management

16. Information Networking Security and Assurance Lab National Chung Cheng University 16 Vendor Independent Testing Lab NSS’s test-bed  P3 1GHz 768 MB RAM running Windows 2000 SP2, FreeBSD 4.4 or Red Hat 6.2/7.1  Ghost image  100M Ethernet with CAT-5, Intel NetStructure 40T routing Switches and Intel auto-sensing 10/100 network cards  IDS installed on a dual-homed PC on each subnet  No firewall used

17. Information Networking Security and Assurance Lab National Chung Cheng University 17 Vendor Independent Testing Lab NSS five types of tests  Attack recognition SAN top 20 and/or ICAT top 10 vulnerability lists  Performance under load Back Orifice ping 64-byte, 1514-byte packets/25,50,75 and 100 percent of network load Adtech AX/4000 Broadband Test System and SmartBits SMB6000

18. Information Networking Security and Assurance Lab National Chung Cheng University 18 Vendor Independent Testing Lab NSS five types of tests  IDS evasion techniques Tools: Fragrouter and whisker  Stateful operation test Tools: stick and snot used to generate false alerts  Host performance Network load, CPU and memory utilizations were monitored

19. Information Networking Security and Assurance Lab National Chung Cheng University 19 Trade Magazine Evaluation Interesting approach  IDSs in the production network of an ISP  Deployed four machines The metrics were accuracy, ease of use, and uptime

20. Information Networking Security and Assurance Lab National Chung Cheng University 20 Conclusion