1. ANCS 2006 Scalable Network-based Buffer Overflow Attack Detection Fu-Hau Hsu Department of Computer Science and Information Engineering National Central University Taoyuan, Taiwan, R.O.C. hsufh@csie.ncu.edu.tw Fanglu Guo Symantec Research Laboratory Cupertino, CA, U.S.A. fanglu_guo@symantec.com Tzi-cker Chiueh Computer Science Department Stony Brook University Stony Brook, NY, U.S.A. chiueh@cs.sunysb.edu

2. ANCS 2006 Virulence of Buffer Overflow Attacks Buffer overflow attack is arguably the most widely used and thus most dangerous attack method used today. Most Internet Worms use it to proliferate themselves. It accounts for more than 50% of all the security vulnerabilities recorded by CERT.

3. ANCS 2006 Proposed Solutions Compiler Transformation Stack Guard, RAD, Address Obfuscation Library Rewriting OS Non-executable Stack Instruction Set Hardware AMD Athlon-64

4. ANCS 2006 Discrepancy between Theory and Practice In theory, these efforts have largely solved the buffer overflow attack problem. In practice, however, new buffer overflow vulnerabilities are still discovered and reported on a routine basis.

5. ANCS 2006 substantial modification substantial resistance Why?

6. ANCS 2006 A Solution to the above Dilemma -- Nebula Nebula A network-based buffer overflow attack detection mechanism Observe the network traffic only to detect BOAs Currently version is developed for Linux paltforms.

7. ANCS 2006 Existing Network-based Intrusion Detection System (NIDS) Misuse intrusion detection Zero-day BOAs Labor-Intensive Solution: automatically signature-generating approaches Anomaly intrusion detection False Positive

8. ANCS 2006 Generalized Signature

9. ANCS 2006 Two Factors for a Successful Buffer Overflow-style Attack A successful buffer overflow-style attack should be able to overflow the right place (e.g. the place to hold a return address with the correct value (e.g. the address of injected code entry point)).

10. ANCS 2006 Non-predicable Offset and Entry Point Address buffer where the overflow start injected code return address offset between the beginning of the overflowed buffer and the overflow target. address of injected code entry point. The offset and the entry point address are non-predicable. They can not decided by just looking the source code or local binary code.

11. ANCS 2006 Non-predicable Offset For performance concerns, most compilers don’t allocate memory for local variables in the order they appear in the source code, sometimes some space may be inserted between them. (Source Code doesn’t help) Different compiler/OS uses different allocation strategy. (Local binaries don’t help) Address obfuscation insert random number of space between local variables and return address. (Super good luck may help)

12. ANCS 2006 Non-predicable Entry Point Address [fhsu@ecsl]# 0xbfffffff system data environment variables argument strings env pointers argv pointers argc webserver –a –b security command line arguments and environment variables Function main()’s stack frame

13. ANCS 2006 Strategies Used by Attackers to Increase Their Success Chance Repeat address patterns. Insert NOP (0x90) operations before the entry point of injected code.

14. ANCS 2006 Indispensable Elements of BO- style Attacks ‘ The Address ’ For buffer overflow attacks, it is the address of the entry point of injected code.

15. ANCS 2006 Linux Process Memory Layout 0xc0000000 0xffffffff kernel address space user stack 8M %esp for Shared libraries, including libc functions brk run-time heap data and code 0x40000000 address space of addresses of injected code and frame pointers (Stack Address Zone)

16. ANCS 2006 Size of Stack Address Zone The default maximum size of a process’s user space stack is 8 Mbytes. However, according to Ditzel et al., the average function frame size is 28 bytes. Therefore, the majority of program are not supposed to use a 2Mbyte stack. In our test, a 8k stack is enough to identify all 10 remote exploit strings.

17. ANCS 2006 Repeating Times and Values of Return Addresses 2k stack --- 0xbffffffff ~ 0xbfffe000

18. ANCS 2006 A Property of Stack Addresses The leading byte of any words that contain a stack address corresponds to a non- printable ASCII character.

19. ANCS 2006 Generalized Signature Signature of a stack smashing buffer overflow attack : If a sub-string of a traffic string could be interpreted as a stack address that repeats 3 or more times, it is alarmed as a buffer overflow attack string.

20. ANCS 2006 Contextual Analysis

21. ANCS 2006 Bypassing Detection Patient attackers could bypass detection based on repeating address signature by repeating addresses no more than 2 times. PS: All the 10 remote exploit code we tested repeat at least 4 times. Attackers repeat the addresses to increase their chance to success. In other words, it is very likely that without the repeat, attackers will fail many times before getting a successful one.

22. ANCS 2006 Unsuccessful Attacks Buffer overflow-style attacks will destroy targeted process’s address space which in turn usually will crash the attacked process. In order to recycle valuable system resources, OS will close the sockets opened by crash processes automatically. On both Linux and Windows, when a program is crashed, the OS will terminate all the program’s pending socket connections by sending out an RST packet to the communicating hosts on its behalf.

23. ANCS 2006 Server Termination Signature After forwarding a sub-string which could be interpreted as a single stack address, Nebula detects that the server closes the TCP connection without sending any data, then the traffic string is deemed as a buffer overflow attack string. Future traffic coming from the same hosts will be blocked or examined thoroughly.

24. ANCS 2006 Will Normal Traffic Behavior the Same Way? HTTP Protocol (RFC 2616) works in the request-reply way. (After the request, there will be a reply before the server close the connection) SMTP protocol (RFC 2821), for e-mail, and FTP protocol(RFC 959) use QUIT command to close a connection. (QUIT can not be interpreted as a stack address.)

25. ANCS 2006 Payload Bypassing

26. ANCS 2006 Payload Bypassing Payload bypassing tries to avoid packet analysis for as much traffic as possible. Because most buffer overflow attacks take place during the exchange of control messages, it is safe to ignore the bulk of data that is downloaded as uninterpreted bytes. For example, in an FTP session, data transferred over the data connection can never be used to mount a buffer overflow attack against the FTP program because the FTP program does not interpret them.

27. ANCS 2006 Internet Traffic Statistic From CacheLogic’s measurement on USA, Europe, and Asia backbone in June 2004, HTTP and P2P packets accounted for more than 70% of the total traffic.

28. ANCS 2006 Percentage of Payload Percentage of payload in the traffic when each of the four protocols that Nebula can recognize is used to transfer files of a total size of 1.22 Gbytes.

29. ANCS 2006 Number of False Positives without Payload Bypassing Number of false positives under the our sample as reported by Nebula. The minimal number of times the attack pattern is repeated is assumed to be 1, 2, 3 or 10, and the stack size tested is 2Mbytes, 16Kbytes, or 8Kbytes. In each entry the left is the number of false positives for RTL attacks, whereas the right is the number of false positives for CI attacks. The sample includes 134966 TCP connections and about 1.582 Gbytes of data.

30. ANCS 2006 Number of False Positives with Payload Bypassing The number of false positives in the test traffic associated with different protocols after applying payload bypassing is negligible even when the attack pattern repetition count is 1.

31. ANCS 2006 Throughput Comparison The throughput of Nebula under a test HTTP connection when different options are turned on. With payload bypassing, Nebula can perform buffer overflow attack detection and still achieve a throughput higher than a generic Linux router.