1. CSC 482/582: Computer Security
Buffer Overflows

2. Topics Buffer Overflows Stack Overflows Heap Overflows Use after Free
Memory Corruption Mitigations
Return Oriented Programming
Writing Secure Code

3. Memory Corruption
Memory corruption occurs when contents of a memory location are modified in a way unintended by the programmer. Modifications can result from
Buffer overflows
Format string attacks (%n)
Use after free
Memory corruption vulnerabilities can result in
Execution of attacker-supplied code.
Attacker hijacking of program control flow.
Alteration of program data.
Information leak.

4. Buffer Overflows
buffer: limited contiguously allocated set of memory.
static: char buffer[32]
dynamic: malloc(), new
What happens when you attempt to access an element beyond the end of the buffer?
Bounds checking prevents such accesses in most languages like Python, Ruby, and Java.
But in C, C++, Objective C, Forth, and assembly large inputs can overflow the buffer, overwriting adjacent data in memory.
The design principles are rooted in simplicity and restrictiveness.
Simplicity lies on many levels. The basic idea is that simpler things have fewer components, so less can go wrong. Further, there are fewer interfaces, so there are fewer entities communicating through the interfaces that can be inconsistent. Finally, they are easier to check, since the mechanism is not complex, and therefore easier to understand. There is also less to check.
Restriction minimizes the number and types of interactions between the entity and other entities. In some circles, an example is the “need to know” principle: only give the entity the information it needs to complete its task. It also should only be able to release information when required to by the goals of the entity. Note this includes writing (integrity), because by altering other entities, the writer can communicate information.

5. C Strings C strings terminated with \0 character.
Many operating systems and software components are written in C
Interfaces inherit semantic “strings end with \0”.
Some components don’t handle \0 embedded in string gracefully, even if programming language can.
Note that UTF-16/UTF-32 include many byte 0s.
Note that \0 takes space – account for it!
Overwriting can create string doesn’t end.
Formal name is NUL character H e l o \0

6. An Example Buffer Overflow
char A[8];
short B=3; A B 3
gets(A); A B o v e r f l w s

7. Out-of-Bounds Read What’s the mistake in this program? int main() {
int array[5] = {1, 2, 3, 4, 5};
printf("%d\n", array[5]); }
Program output:
> gcc -o buffer buffer.c
> ./buffer

8. Out of Bounds Write Writing beyond the buffer: Program output:
int main() {
int array[5] = {1, 2, 3, 4, 5};
int i;
for( i=0; i <= 255; ++i )
array[i] = 41; }
Program output:
> gcc -o bufferw bufferw.c
> ./bufferw
Segmentation fault (core dumped)

9. What happens when a buffer overflows?
What happened to our buffer overflow?
Overwrote memory beyond buffer with 41.
Program crashed with Segmentation fault.
Directly or indirectly accessed unmapped memory.
Creates a page fault exception.
OS does not find mapping on page table.
OS sends segmentation fault signal to process.
Do overflows always produce a crash?
Unintentional overflows usually do, but
Attackers will restrict writes to mapped pages.

10. History of Overflows
1972: Computer Security Technology Planning Study.
1988: Morris Worm spreads with fingerd overflow. CERT formed.
1989: CERT published CA for BSD 4.3 passwd overflow.
1995: Thomas Lopatic posts info on httpd overflow to Bugtraq.
1996: Aleph One publishes “Smashing the Stack for Fun and Profit”
1997: Solar Designer describes return-to-libc technique.
1999: Compiler-based mitigations proposed.
2001: Code Red worm exploits overflow in IIS.
2003: Slammer worm infects every vulnerable server in an hour.
2004: AMD and Intel add non-executable bit in page tables.
2008: Twilight hack unlocks Wii consoles.
2010: Android 2.3 adds support for non-exec stack.
2013: Unicode SSIDs crash iPhones.
2015: GHOST overflow impacts glibc.
Twilight Hack creates an absurdly-long horse name for “The Legend of Zelda: Twilight Princess” that includes a program

11. Process Memory Map Text (compiled program code) Initialized
Lower-numbered
addresses
Higher-numbered
Warning: Some
diagrams elsewhere
show lower-numbered
addresses at the bottom
Text (compiled program code)
Often
read-
only
Stack (procedure/ method calls)
Heap
(dynamically allocated)
Heap grows, e.g.,
due to “new” or malloc()
Stack grows, e.g.,
due to procedure call
Stack pointer (SP)
(current top of stack)
This diagram shows
how stacks grow on
Intel x86s & others;
some grow other way.
Multi-threaded programs
have multiple stacks
Heap pointer
Initialized
global “data”
Uninitialized
Used
for global
constants
& variables
Set on
code
load

12. Memory Layout Example /* data segment: initialized global data */
int a[] = { 1, 2, 3, 4, 5 };
/* bss segment: uninitialized global data */
int b;
/* text segment: contains program code */
int main(int argc, char **argv) /* ptr to argv */ {
/* stack: local variables */
int *c;
/* heap: dynamic allocation by new or malloc */
c = (int *)malloc(5 * sizeof(int)); }

13. Abstract data type “Stack”
“Stack”: Abstract CS concept
“A stack of objects has the property that the last object placed on the stack will be the first object removed. This property is commonly referred to as last in, first out queue” (LIFO).
Minimum stack operations:
PUSH: Add an element to the top of the stack
POP: Removes the last element at the top of the stack (returning it) and reduces stack size by one.

14. “Stack” in process memory map
Memory area set aside to implement calls to a procedure/function/method/subroutine
For now we’ll use these terms interchangeably
In C the term is “function”
Stack is used to implement control flow
When you call a procedure, where it “came from” is pushed on stack
When a procedure returns, the “where I came from” is popped from stack; system starts running code there
Stack also used for other data (in many cases)
Parameters passed to procedures
Procedure local variables
Return values from procedure

15. Call Stack Layout Unallocated Stack Frame b() { for b() … } for a()
Low Memory
b() { … }
a() {
b();
main() {
a();
Unallocated
Stack Frame
for b()
for a()
for main()
High Memory

16. What is a Stack Frame? Block of stack data for one procedure call.
Frame pointer (FP) points to frame:
Use offsets to find local variables.
SP continually moves with push/pops.
FP only moves on function call/return.
Intel CPUs use %ebp register for FP.

17. Why use stacks for function calls?
First compiled languages (e.g., FORTRAN) did not use stacks
Call data stored with fn where program “came from”
Result: Procedures could not call themselves, directly or indirectly, as that would overwrite stored information.
If functions can arbitrarily call other functions
Need to store old state so can return back
Need dynamic allocation for call (frame) sequences
Stack is flexible & efficient

18. CPUs track two stack values
Stack pointer (SP): Value of “top” of stack
Where last data was stored on stack, possibly +/- 1 depending on architecture conventions
Modified when data pushed/popped
May even be modified during expression calculation
Frame pointer (FP): Value of “this frame”
Simplifies accessing parameters & local variables
Points inside stack to where “this procedure” starts
Modified on entry/exit of a procedure

19. Accessing the Stack Pushing an item onto the stack.
Copy 4 bytes of data to stack.
Decrement SP by 4.
Example: pushl $12
Popping data from the stack.
Copy 4 bytes of data from stack.
Increment SP by 4.
Example: popl %eax
Retrieve data without pop: movl %esp, %eax

20. Calling a procedure f(1,2,3); } Given this C program: void main() {
The invocation of f() might generate assembly:
pushl $3 ; constant 3
pushl $2 ; Most C compilers push in reverse order
pushl $1
call f
“call” instruction pushes instruction pointer (IP) on stack
In this case, the position in “main()” just after f(…)
Saved IP named the return address (RET)
CPU then jumps to start of “function”

21. Stack: After push of value 3
Lower-numbered
addresses
Higher-numbered
addresses
Stack pointer (SP)
(current top of stack) 3

22. Stack: After push of value 2
Lower-numbered
addresses
Stack pointer (SP)
(current top of stack) 2
Higher-numbered
addresses 3
Stack grows, e.g.,
due to procedure call

23. Stack: After push of value 1
Lower-numbered
addresses
Stack pointer (SP)
(current top of stack) 1 2
Higher-numbered
addresses 3
Stack grows, e.g.,
due to procedure call

24. Stack: Immediately after call
Lower-numbered
addresses
Stack pointer (SP)
(current top of stack)
Return address in main() 1 2
Higher-numbered
addresses 3
Stack grows, e.g.,
due to procedure call

25. In the assembly above, “;” introduces a comment to end of line
Function prologue
Imagine f() has local variables, e.g. in C:
void f(int a, int b, int c) {
char buffer1[5];
char buffer2[10];
strcpy(buffer2, "This is a very long string!!!!!!!"); }
Typical x86-32 assembly on entry of f() (“prologue”):
pushl %ebp ; Push old frame pointer (FP)
movl %esp,%ebp ; New FP is old SP
subl $20,%esp ; New SP is after local vars
; “$20” is calculated to be >= local var space
In the assembly above, “;” introduces a comment to end of line

26. Stack: Immediately after call
Lower-numbered
addresses
Stack pointer (SP)
(current top of stack)
Return address in main() 1 2
Higher-numbered
addresses 3
Stack grows, e.g.,
due to procedure call

27. Stack: After prologue Local array “buffer2” Local array “buffer1”
Lower-numbered
addresses
Stack pointer (SP)
(current top of stack)
Local array “buffer2”
Local array “buffer1”
Frame pointer (FP) –
use this to access
local variables &
parameters
Saved (old) frame pointer
Return address in main() 1 2
Higher-numbered
addresses 3
Stack grows, e.g.,
due to procedure call

28. Stack: Overflowing buffer2
Lower-numbered
addresses
Stack pointer (SP)
(current top of stack)
Local array “buffer2”
Overwrite
Local array “buffer1”
Frame pointer (FP) –
use this to access
local variables &
parameters
Saved (old) frame pointer
Return address in main() 1 2
Higher-numbered
addresses 3
Stack grows, e.g.,
due to procedure call

29. Effect of Overflowing Buffer2
Overwrites whatever is past buffer2!
As you go further, overwrite higher addresses
Impact depends on system details
In our example, can overwrite:
Local values (buffer1)
Saved frame pointer
Return value (changing what we return to)
Parameters to function
Previous frames

30. Common buffer overflow attack
Send too large data as user input.
Overlarge data overwrites buffer
Modifies return value, to point to something the attacker wants us to run
Maybe with different parameters, too
On return, runs attacker-selected code
But it gets worse…

31. Shellcode Injection
Attacker can also include machine code that they want us to run.
If they can set the “return” value to point to this malicious code, on return the victim will run that code
Unless something else is done
Significant portion of “Smashing the Stack” paper describes how to insert such code.

32. Stack: Overflow with Shellcode
Lower-numbered
addresses
Stack pointer (SP)
(current top of stack)
Local array “buffer2”
Malicious code
Local array “buffer1”
Frame pointer (FP) –
use this to access
local variables &
parameters
Saved (old) frame pointer
Return address in main()
Ptr to malicious code 1 2
Higher-numbered
addresses 3
Stack grows, e.g.,
due to procedure call

33. Stack: Shellcode + NOP Sled
Lower-numbered
addresses
Stack pointer (SP)
(current top of stack)
Local array “buffer2”
NOP sled: \x90\x90\x90\x90\x90….
NOP sleds let attacker jump anywhere to attack; real ones often more complex (to evade detection)
Shellcode: \xeb\x1f\x5e\x89\x76\x08\x31\xc0\x88\x46\x07\x89\x46\x0c\xb0\x0b\x89\xf3\x8d\x4e\x08\x8d\x56\x0c\xcd\x80\x31\xdb\x89\xd8\x40\xcd\x80\xe8\xdc\xff\xff\xff/bin/sh
Local array “buffer1”
Frame pointer (FP) –
use this to access
local variables &
parameters
Shellcode often has odd constraints, e.g., no byte 0
Saved (old) frame pointer
Return address in main()
Ptr to malicious code 1 2
Higher-numbered
addresses 3
Stack grows, e.g.,
due to procedure call

34. Shellcode
Shellcode is a small piece of machine code inserted into a program by exploiting a vulnerability.
Called shellcode since it is often used to start a command shell under control of attacker.
Example shellcode
Remote shell (like ssh)
Reverse shell (opens connection to attacker server)
Remote desktop (RDP, VNC, etc.)
Downloader (installs remote control tools)

35. Off-by-One Overflows
Even 1 byte overflow can allow attacker to control execution by overwriting low order byte of FP.
Figure 5.10 from Buffer Overflow Attacks.

36. Heap Overflows Programs allocate memory on heap via
C: malloc() and free()
C++: new and delete
To run shellcode, attack overwrites function pointer
Function pointers in C.
C++ methods implemented as function pointers.
C++ exceptions result in following a function pointer.

37. Other Function Pointer Targets
C and C++ standards
atexit() functions
longjmp() buffers
C++ exceptions
Linux
Global Offset Table in ELF binaries.
.dtors section in gcc binaries.
Windows
Vectored exception handling (VEH)
Structured exception handling (SEH)

38. Memory Allocation Memory allocators typically
Allocate memory in chunks.
Chunks stored in a linked list of bins.
Chunks contain header, pointers, and data.
Each bin contains chunks of a specific size.
Final chunk contains free space information.
Memory corruption attacks target
Header information, and
Pointers.

39. Malloc Data Structure
Scot Hand, Heap Based Exploitation class presentation at UTD.

40. Malloc Example
char *a, *b, *c; a = malloc(32) b = malloc(32); c = malloc(32); … free(c) free(b) free(a)

41. Malloc Example Memory Layout

42. Malloc Layout after Free

43. Overflowing Dynamic Buffers
Figure 6.3 from Buffer Overflow Attacks

44. Overwriting vtable Pointers
Figure 6.4 from Buffer Overflow Attacks

45. free() vulnerabilities
To de-allocate memory, free(void *) must
Use pointer to find chunk to de-allocate.
Modify pointers in chunk header to remove chunk from the list of in-use chunks.
Loop: if adjacent chunk is free, combine chunks to avoid memory fragmentation.
To exploit free(), an attacker can
Overwrite adjacent chunk metadata.
Cause free() to overwrite a specific word in memory with attacker specified data.
Attacker will choice location to be that of a pointer (lib function, vtable ptr, ret addr, etc.) to control execution.

46. Double free() vulnerability
Program free()s chunk twice, so chunk appears twice in the free list.
Program allocates memory, reusing part or all of chunk.
malloc() interprets first bytes as free list header.
Program interprets first bytes as data.
Write to allocated chunk, changing first bytes.
Program allocates, attempts to reuse chunk.
malloc() writes to location specified in first bytes, treating bytes as free list pointer, but those bytes now point to any location specified by attacker.

47. Non-executable Stack
Memory protection prevents code on the stack from being executed.
Adds NX permission bit to page tables.
Required CPU and OS modifications.
Limitations
Some applications need to execute code on the stack (JIT compilers for Java, JavaScript).
Attackers can target other areas of memory.
Buffer overflows can result in remote code execution without running attacker generated shellcode.

48. Address Space Randomization
ASLR randomly changes code/data locations to prevent memory exploits.
Problems
Code must be compiled as Position Independent Executables (PIE) to relocate code segment at cost of a 10% performance penalty.
Memory corruption can result in information leakage of address layout.
Attackers can brute force 32-bit ASLR with many copies of shellcode (heap-spraying).

49. Canary Defenses Compiler changes calling convention in two ways:
Adds a canary word to the stack (“canary in a coal mine”)
Verifies presence of canary word before executing the ret instruction to return to address on stack.
Protects against stack smashing since
Overflow would have to overwrite canary to reach return value.
If canary is chosen randomly, attacker cannot know what to overwrite to that memory location.
Does not protect against indirect memory writes.

50. Canary Stack Layout old frame param1 param2 old PC canary word old FP
Frame Pointer
Stack Pointer
old frame
param1
param2
old PC
canary word
old FP
local vars
FP still points at new frame.
SP points at current top of stack, moving with local var allocation/deallocation.

51. Encoding Function Pointers
Store encrypted function ptr in program.
Decrypt function pointer before use.
Encrypted pointers overwritten by attacker will decrypt to random address, stopping exploit.
Must encrypt all function ptrs to be secure.
Windows API used in OS, avail for users
EncodePointer()
DecodePointer()
Proposed for C11 standard, but not added.

52. Return-Oriented Programming
ROP is an exploit technique in which the attacker gains control of the call stack and uses it to execute small pieces of code, called “gadgets.”
Attacker uses existing code.
Bypasses NX defense, since no new code executed.
Gadgets are typically found in shared libraries
Gadgets can be entire functions.
Gadgets can be fragments of code that end in a ret instruction (even unintentional instructions.)
Attacker controls order of execution and parameters by placing data on call stack.

53. Normal Programming
Instruction pointer (%eip) determines which instruction to fetch and execute.
CPU automatically increments %eip to next instruction.
Control flow by modifying %eip.

54. Return Oriented Programming
Stack pointer (%esp) determines which instruction sequence to fetch and execute.
Move to next sequence after ret instruction.

55. Finding Gadgets Intel x86 assembly is dense As a result
Instructions are of variable length.
Instruction pointer can point to any location.
As a result
Any byte c3 is a ret instruction.
By pointing %eip into the middle of a long instruction, part of that instruction can be executed as a different instruction or set of instructions.

56. Unintended Instructions Example

57. Writing Overflow Free Code
Know which functions are
Unsafe: guaranteed buffer overflow.
Safe: no overflow if arguments correct.
Know how to use safe functions securely.
Do your own bounds checking.
Be careful to avoid integer overflows.
Check for memory errors
Security-oriented static analysis like Fortify.
Memory dynamic analysis: Valgrind, Purify.

58. Many C fns don’t check bounds
gets(3) – reads input without checking. Don’t use it!
strcpy(3) – strcpy(dest, src) copies from src to dest
If src longer than dest buffer, keeps writing!
strcat(3) – strcat(dest, src) appends src to dest
If src + data in dest longer than dest buffer, keeps writing!
scanf() family of input functions – many dangerous
scanf(3), fscanf(3), sscanf(3), vscanf(3), vsscanf(3), vfscanf(3)
Many options don’t control max length (e.g., bare “%s”)
Many other dangerous functions, e.g.:
realpath(3), getopt(3), getpass(3)
streadd(3), strecpy(3), and strtrns(3)
It’s not just functions; ordinary loops can overflow.

59. gets() Never use gets(). Avoiding unsafe functions not enough:
You can replicate this problem yourself with a getchar() loop where you copy chars to buffer w/o checking.

60. strcpy()
strcat() same problem, has strncat() variant

61. sprintf()
Can’t simply switch to printf() because of format string vulnerabilities discovered in 2000, and you’re not always printing the string immediately in any case.

62. Bounded Function Pitfalls
Destination buffer overflows because bound depends on size of source data, not destination buffer.
Destination buffer left without null terminator, often as result of off-by-one error.
Destination buffer overflows because its bound is specified as the total size of the buffer, rather than space remaining.
Programs writes to arbitrary location in memory as destination buffer is not null-terminated and function begins writing at location of first null in destination buffer.

63. C11 Annex K Bounds Checking
C11 standard adds bounds-checking interfaces
Creates “safer” versions of C functions
Limits lengths of results
Example: strcpy_s(s1, s1max, s2);
Copies s2 to s1.
s1max > strnlen_s(s2, s1max)
Does not automatically resize.
On error, calls runtime-constraint handler function, controlled by set_constraint_handler_s(). This handler can permit returns
If it returns, returns 0 if ok, nonzero if a constraint failed
Not universally available. Provided by slibc.

64. Safe String Libraries strlcat() and strlcpy() C++ std::string library
BSD-licensed open source routines
C++ std::string library
Dynamically-sized strings
SafeStr library provides safestr_t objects
Dynamically-sized
Cast to (char *) for read-only purposes only
Microsoft’s strsafe.h
C++ can be used with C-style strings in dangerous ways (see example in a few slides.)

65. Safe String Libraries UNIX Libraries Windows Libraries C C Bstrlib
MT-Safe
SafeStr
strlcpy(), strlcat()
Vstr
C++
std::string (STL)
Windows Libraries C
Safe CRT
strlcpy(), strlcat()
StrSafe
C++
CString (MFC)
Safe C++
std::string (STL)
C++ can be used with C-style strings in dangerous ways (see example in a few slides.)

66. strlcpy() and strlcat()
size_t strlcpy (char *dst, const char *src, size_t size);
size_t strlcat (char *dst, const char *src, size_t size);
Size is max size of dest buffer (not maximum number of chars to copy), including NULL.
Destination buffer always NULL terminated
Return how much space would be required in destination buffer to perform operation.
BSD-style open source license.
strlcat() and strlcpy() exist.

67. Character Sets
Characters represented using encoding forms that map code points to printable chars.
Fixed Width
ISO
UTF-32
Variable Width
UTF-8 (most Internet protocols, Python, Ruby)
UTF-16 (Java, .NET)
Character
Encoding
Code Point s
ISO
UTF-8 73 ÿ FF
C3 BF

68. Wide Characters C/C++ Regular and Wide String Types
char and string use 1-byte characters.
wchar_t and wstring use wide (2 or 4 byte).
String Sizing Issues
Length of string ≠ size of array for wide chars.
Use strlen() for char based strings.
Use wcslen() for wchar_t based strings.

69. C++ Dangers Using C-style strings with cin
char username[16];
cin >> username;
The [] operator does not perform bounds checking
Converting from C++ to C-style strings
string::data() output is not NULL terminated
string::c_str() ouput is NULL terminated

70. Key Points Memory corruption vulnerabilities can result in
Execution of attacker-supplied code.
Attacker hijacking of program control flow.
Alteration of program data.
Information leak.
Memory Corruption Attack Types
Buffer overflows: stack and heap
Format string attacks (%n)
Use after free()

71. Key Points (2) Mitigations Countering Mitigations Writing Secure Code
Non-executable Stack (OS, hardware)
Address Space Layout Randomization (OS)
Stack Canaries (compiler)
Countering Mitigations
Brute force (heap spraying)
Return Oriented Programming
Writing Secure Code
Avoiding unsafe functions.
Performing bounds checking.

72. References
Aleph Null, “Smashing the Stack for Fun and Profit,” Phrack 49, 1996.
Erik Buchanan et al., “Return-Oriented Programming: Exploits Without Code Injection”,
Johnathan Bartlett, Programming from the Ground Up, Bartlett Publishing, 2004.
James C. Foster et al., Buffer Overflow Attacks, Syngress, 2005.
Scot Hand, Heap Based Exploitation,
Koziol et al., The Shellcoder’s Handbook: Discovering and Exploiting Security Holes, Wiley, 2004.
Haroon Meer, “History of Memory Corruption Attacks”,
Tom Plum, “The New C Standard Explored”, Dr. Dobb’s Journal, 2012.
Robert C. Seacord, Secure Coding in C and C++ 2nd ed., Addison-Wesley, 2013.
Laszlo Szekeres et al. "Eternal war in memory." Proceedings of the 34th IEEE Symposium on Security and Privacy (SP’13), San Francisco, CA
John Viega and Gary McGraw, Building Secure Software, Addison-Wesley, 2002.