Design of a Framework for Testing Security Mechanisms for Program-Based Attacks Ben “Security” Breech and Lori Pollock University of Delaware

Motivation o Unsafe programming practices –Exploited by malicious input (program-based attack) o Vulnerabilities difficult to find and fix –Expensive (time, resources required) –Often wait until exploit is found o Mechanisms proposed to protect programs –Poorly tested –Low confidence in security protection

Testing Security Mechanisms o Testing is often poor and non-systematic –Current process: Find program with vulnerability and known exploit. Apply the mechanism and try the exploit. –Finding test subjects can be difficult o Need a framework that provides a systematic, and automatic method of testing security mechanisms –More thorough testing of mechanisms

Program-Based Attacks o Malicious attacks initiated as input –Usually buffer overflow o Examples –Stack smashing –Function pointers –Heap variables

Example Attack int proc_inp (int a, int b, int c) { char buf1 [3]; int z; char buf2 [5]; gets (buf2) …. } 42 Input: ABCDEFGHIJKLMNOPQRST c -> 6 7 0x1200 b -> a -> return address -> saved fp -> buf1 [] -> 0x150 z -> buf2 [] -> QRST MNOP JKL FGHI ABCDE Caller AR proc_inp AR

Example Mechanism: RAD A() { char s [5]; gets (s); B(s) … ret } B(char *s) { C(s) … ret } C(char *s) { … ret } A AR: 0x1200 C AR: 0x1280 B AR: 0x1250 0x1200 0x1250 0x1280 Call Stack: RAD Stack: (Chiueh and Hsu, ICDCS 2001) Input: ABC

Our Framework… o Present the design of a framework to test security mechanisms –Enables systematic and automatic testing o Key insight: use dynamic compiler to simulate attacks

Framework Design

Framework Key Components

Framework Requirements o General -- support different languages, vulnerabilities and mechanisms o Systematic -- insert attacks at appropriate points o Automatic -- little user interaction o Support testing of evolving programs o Robust -- few false positives o Low overhead -- both space and time

The Testing Process A AR: 0x1200 A AR: 0x5000 C AR: 0x1280 B AR: 0x1250 0x1200 0x1250 0x1280 Call Stack: RAD Stack: B AR: 0x5000 C AR: 0x5000 A() { char s [5]; gets (s); B(s) … ret } B(char *s) { C(s) … ret } C(char *s) { … ret } Input: ABC

Prototype Implementation Use the DynamoRIO 1 dynamic compiler 1 (Bruening et al. CGO 03) Native Binary Code Initialization Basic Block Construction Initialization Basic Block Execution Exit Analysis or Optimization Cleanup And Exit DynamoRIO (simplified view) Client Module

Implementation Issues o Simulation of attacks -- may not perfectly mimic all effects of attack o Efficiency -- overhead tradeoff o Monitoring -- what to look for? o Key component goals -- tradeoff? –No extra code compiled in –All possible attack points tested o Automatically specifying attacks

Summary and Current Status o Presented design of framework for testing security mechanisms for program-based attacks o Proof of concept implementation