1. Breno de MedeirosFlorida State University Fall 2005 Buffer overflow and stack smashing attacks Principles of application software security

2. Breno de MedeirosFlorida State University Fall 2005 Buffer overflows  One of the most common vulnerabilities in software  Particularly problematic when present in system libraries and other code that runs with high execution privileges.

3. Breno de MedeirosFlorida State University Fall 2005 How it works  Application reserves adjacent memory locations (buffer) to store arguments to a function, or variable values.  Attacker gives an argument too long to fit in the buffer.  The application copies the whole argument, overflowing the buffer and overwriting memory space.  If the conditions are “just right” this will enable to attacker to gain control over the program flow and execute arbitrary code, with the same privileges of the original application.

4. Breno de MedeirosFlorida State University Fall 2005 Stack smashing  Function (sub- routine) calls results in an activation frame being pushed onto a memory area called the stack. function arguments return address previous frame pointer local variables local buffer variables Direction of stack growth

5. Breno de MedeirosFlorida State University Fall 2005 Memory management  The stack, which contains activation frames, starts at the highest memory address allocated for the process, and grows downwards  Variable-length data (e.g., strings) that are read dynamically, are kept in the heap, which grows upwards  This arrangement maximizes flexibility of virtual memory management Stack Heap unitialized variables initialized variables code instructions Direction of stack growth Direction of heap growth

6. Breno de MedeirosFlorida State University Fall 2005 How to smash  Give application a very long string with malicious code  The string length, being much larger than the space allocated in the heap (buffer size declaration) causes the heap to overflow into the stack and overwrites the return address  The return address now points to the beginning of the malicious code function arguments Return address (overwritten with entry address of malicious code) Previous frame pointer (overwritten w/ malicious code) local variables (overwritten w/ malicious code) local buffer variables (overwritten w/ malicious code) Direction of stack growth

7. Breno de MedeirosFlorida State University Fall 2005 Canary Guards  Like the legendary canary-in-the-mine, it detects stack smash attacks.  Inserts a “Canary value” just below the return address (Stack Guard) or just below the previous frame pointer (Stack Smashing Protector). This value gets checked right before a function returns.

8. Breno de MedeirosFlorida State University Fall 2005 SSP  Prevents overflow of local non-buffer variables  Canary value checking only takes place at return time, so other attacks possible function arguments return address previous frame pointer local buffer variables local non-buffer variables Direction of stack growth Canary value

9. Breno de MedeirosFlorida State University Fall 2005 Alternatives to canaries  Use a compiler that does full bounds checking, i.e., makes sure that the code always allocate enough memory for arguments  Like SSP, code has to be re-compiled with this compiler  Significant performance penalty (Java/C)

10. Breno de MedeirosFlorida State University Fall 2005 Static analysis  Use a code analyzer to detect buffer overflows  Since checking that arbitrary code does not overflow is an undecidable problem, the code must be annotated in order for this to work  Depends on programmer expertise (costly)  Some common and useful programming techniques are prohibited (performance and engineering costs)  Advantage is that the compiled code does not suffer from performance deterioration

11. Breno de MedeirosFlorida State University Fall 2005 Safe libraries  Many vulnerabilities in code are due to unsafe use of system libraries  An alternative is to install a kernel patch that dynamically substitutes calls to unsafe library functions for safe versions of those  Not possible for closed-source systems such as MS operating systems

12. Breno de MedeirosFlorida State University Fall 2005 Memory address randomization  Patch at the kernel level, changing the memory mapping  Small performance penalty, by extra memory lookups (actually, extra cache lookups)  Makes it very difficult to perform a useful buffer overflow  However, unlike some other strategies, does not improve robustness (liveness) properties