Buffer Overflow Detection Stuart Pickard CSCI 297 June 14, 2005

Papers “StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks.” Crispin Cowan, Calton Pu, Dave Maier, Heather Hinton, Peat Bakke, Steve Beattie, Aaron Grier, Perry Wagle, and Qian Zhang. 7th USENIX Security Symposium, January 1998, San Antonio, TX. “Buffer Overflows: Attacks and Defenses for the Vulnerability of the Decade.” Crispin Cowan, Perry Wagle, Calton Pu, Steve Beattie, and Jonathan Walpole. SANS 2000, Orlando FL, March “Type-Assisted Dynamic Buffer Overflow Detection.” K.S.Hlee and J.S.Chapin. 11th Annual USENIX Security Symposium 2002.

The First Two Papers G1 was published in 1998 G2 was published in 2000 Both papers focus on Static Buffer Overflow detection and prevention in the compilation stage. Lead to the development of software for the Immunix project. The project then became Immunix Corporation. On May 10, 2005 Immunix was bought by Novell and the products are being sold under the name of AppArmor

Quote from Press Release “The new product, to be called Novell® AppArmor powered by Immunix,TM protects both the Linux operating system and applications from external or internal attacks, viruses, and malicious applications.”

Buffer Overflow life Cycle 1. A malicious user finds the vulnerability in a highly privileged program. 2. The user might be able to gain access to the system by overflowing a buffer and pushing code into the return address space of a function call. 3. Fixes are then made by patching the code to prevent future attacks.

4 basic approaches to defending against buffer overflow attacks 1. Write Code to fully check array bounds 2. Non-Executable Buffers – prevent the data segment of the victim program’s address space from being executed 3. Array Bounds Checking – Check all accesses and writes to an array. (Pruify) 4. Perform integrity checks on code pointers before dereferencing them to detect an attack (not as strong as the others)

Problem with patches Fixes to buffer overflows with patches are attacking the problem at the source. The first two papers suggest solving the problem of buffer overflows by protecting the stack (The Destination of the Attack)

Brief Review of a Buffer Overflow (Stack Smashing) The injected attack code typically spawns a shell with root privileges. StackGuard will prevent the injected attack code from executing. The next slide discusses their method. If a program is written with a buffer overflow vulnerability, then the attacker can crash Stack Guard

StackGuard StackGuard prevents change to the return address of a function on the stack by either preventing change to the address or by preventing the write to the return address. StackGuard is more effective when the return address cannot be altered, however there is more overhead. StackGuard can run in both modes.

How StackGuard is Implemented The return address is unaltered IIF the canary word is unaltered. Reason: This only really works for buffer overflow attacks since the attacker needs to write in a linear sequential manner. Thus, if the return address is modified, then the canary word is also modified. StackGuard is implemented by modifying the open source gcc compiler functions: function_prologue and function_epilogue. The modifications include modified code that emits a canary word to the stack and checks that the canary word has not been modified before the function return.

StackGuard continued total changes are under 100 extra lines of code to the gcc compiler. Canary values need to be random StackGuard Example:

MemGuard: Preventing return address changes MemGuard will protect the return address by making it read-only when the function is called. It will then un- protect the return address when the function finishes and needs to return The protection and un-protection occur in gcc’s function_prologue and function_epilogue.

StackGuard Canary version is more efficient, however, MemGuard is more secure. Immunix project solution: Run StackGuard in Canary version until a treat is detected, then run with MemGuard.

Results

Final thoughts on StackGuard StackGuard not only patches a broad range of existing faults, it patches faults that have not been discovered. It turns the problem of attackers gaining root access to a problem of mild degradation-of-service attacks (switch from Canary to MemGuard mode) This software gives software developers more time to fix these problems. Compare StackGuard to Purify and other array bounds checking for C. In regards to level of overhead, we have Canary Mode as most efficient then MemGuard, then Purify, then other array bounds checking implementations.

PointGuard - Pointer Integrity Checking In the second paper, not only is StackGuard discussed, but the new (as of 2000) patch to the gcc compiler is PointGuard PointGuard is a generalization of StackGuard which places “canaries” next to all code pointers (function pointers) As of writing this paper, PointGuard is not past the prototyping stage. PointGuard will include a special canary storage class that forces canary protection onto arbitrary variables.

Questions? StackGuard – In MemGuard mode or Canary Mode PointGuard Immunix Novell AppArmor

Third Paper – Type-Assisted Dynamic Buffer Overflow Detection This paper first covers many of the other buffer overflow prevention techniques. Most notably, they describe StackGuard and how it is vulnerable to some attacks. They claim that StackGuarded software is vulnerable to attacks exploiting code pointers other than the return address. With this vulnerability StackGuard can be bypassed. They also claim that StackGuard can be bypassed by exploiting the heap memory objects.

Their Solution Enable range checking on buffers at runtime by implementing an intermediary step during compilation that emits an additional data structure into the binary file. This structure describes the types of automatic buffers and static buffers. They generate a type table – a data structure that associates the address of each function with the information of the function’s automatic buffers. They implemented a prototype by extending the GNU C compiler on Linux.

Their Solution continued... The source files are not modified. Just the separate type tables are created. Range checking is done with the following steps: 1. Locate the stack frame of the buffer by chasing down the saved frame pointer. 2. Retrieve the return address of the next stack frame to find out who allocated the stack frame. 3. Locate the function who allocated the stack frame by comparing the return address with function addresses in the type table. 4. Locate the buffer of the function by comparing the buffer address with offsets in the table + frame pointer value, 5. The size of the buffer (or the size of a field if it is a struct variable) is returned

Limitations to their solution There are two cases where they cannot determine the size of automatic buffers. 1. stack buffers dynamically allocated with alloca(); 2. variable-length automatic arrays

Results: C-library string functions

Conclusions The results from the last paper point to the fact that it might not be practical to keep track of buffer ranges during runtime. However, certain vulnerabilities that exist with the StackGuard solution, are prevented with Type-Assisted Dynamic Buffer Overflow Detection

Conclusions..... The solution to buffer overflow attacks most likely lies in a combination of these techniques. Also, maybe it is unlikely that we can totally be guaranteed prevention of these kind of attacks.

References: “StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks.” Crispin Cowan, Calton Pu, Dave Maier, Heather Hinton, Peat Bakke, Steve Beattie, Aaron Grier, Perry Wagle, and Qian Zhang. 7th USENIX Security Symposium, January 1998, San Antonio, TX. “Buffer Overflows: Attacks and Defenses for the Vulnerability of the Decade.” Crispin Cowan, Perry Wagle, Calton Pu, Steve Beattie, and Jonathan Walpole. SANS 2000, Orlando FL, March “Type-Assisted Dynamic Buffer Overflow Detection.” K.S.Hlee and J.S.Chapin. 11th Annual USENIX Security Symposium “Buffer Overflow demo: Embry-Riddle, NSF Scholarships for Service Grant.” Susan Gerhart, Jan Hogle, Jedidiah Crandall

Discussion