Experimental Computer Systems Lab A Binary Rewriting Defense Against Stack-based Buffer Overflow Attacks Manish Prasad, Tzi-cker Chiueh SUNY Stony Brook

Experimental Computer Systems Lab Roadmap Binary Translation Buffer Overflow and Return Address Defense (RAD) Static BT and RAD Experimental Results Conclusion

Experimental Computer Systems Lab Binary Translation Motivation To do things without access to source Traditional Applications Legacy Code Migration (HP Aries, UQBT) Program Optimization (Etch, IBM BOA) Approaches (and their limitations) Static – Lacks transparency, accuracy Dynamic – Less efficient

Experimental Computer Systems Lab Binary Translation and Software Security Legacy applications from outside vendor

Experimental Computer Systems Lab Binary Translation and Software Security Application A Vendor X NO SOURCE CODE CERT Advisory Application A Vendor X VULNERABILITY !!!

Experimental Computer Systems Lab Binary Translation and Software Security CERT Advisory Application A Vendor X VULNERABILITY !!!

Experimental Computer Systems Lab Has it been done yet ?? Dynamic Translation DynamoRIO [MIT] LibVerify [Bell-Labs] Hardware Support [UIUC]

Experimental Computer Systems Lab Why Static Binary Translation (SBT) ? Inferring legacy program behavior Application-specific security policies Use Static Analysis if you can Butler Lampson, Hints for Computer System Design

Experimental Computer Systems Lab Goals How far can we go with pure static BT ? Buffer Overflow protection Widespread Simple yet covers most SBT issues Foundation for general Win32/PE instrumentation framework

Experimental Computer Systems Lab Contributions Comprehensive Treatment of Static BT Why and where it fails ? Prototype implementation incorporating static BT state-of-the-art High disassembly precision Exhaustive experiments with several commercial grade Windows applications

Experimental Computer Systems Lab Buffer Overflow Attack 0 4G Stack Growth Stack Evolution on a Function Call Buffer Overflow and RAD

Experimental Computer Systems Lab Buffer Overflow Attack Function Arguments caller Stack Evolution on a Function Call Buffer Overflow and RAD 0 4G Stack Growth

Experimental Computer Systems Lab Buffer Overflow Attack Function Arguments Return Address CALL Stack Evolution on a Function Call Buffer Overflow and RAD 0 4G Stack Growth

Experimental Computer Systems Lab Buffer Overflow Attack Function Arguments Return Address Old Frame Pointer Local Variables Local Buffer Other Local Variables callee Stack Evolution on a Function Call Buffer Overflow and RAD 0 4G Stack Growth

Experimental Computer Systems Lab Buffer Overflow Attack Function Arguments Return Address Old Frame Pointer Local Variables Local Buffer Other Local Variables target Stack Evolution on a Function Call Buffer Overflow and RAD 0 4G Stack Growth

Experimental Computer Systems Lab Buffer Overflow Attack Function Arguments Return Address Old Frame Pointer Local Variables Local Buffer Other Local Variables A A A A Exp Attack Buffer A = addr of exploit Exp = Exploit Code Unbounded Buffer Copy Buffer Overflow and RAD 0 4G Stack Growth

Experimental Computer Systems Lab Buffer Overflow Attack Exploit Code Address of Exploit Other Local Variables return Overflow Aftermath Buffer Overflow and RAD 0 4G Stack Growth

Experimental Computer Systems Lab Return Address Defense (RAD) Arguments caller Return Addr CALL Stack Ret Addr Copy Return Address Repository (RAR) Prologue Return Address Save Buffer Overflow and RAD

Experimental Computer Systems Lab Return Address Defense (RAD) Arguments Return Addr Stack Ret Addr Copy Return Address Repository Local Variable Space callee Buffer Overflow and RAD

Experimental Computer Systems Lab Return Address Defense (RAD) Ret addr corrupted Stack Ret Addr Copy Return Address Repository Address Of Exploit Code Unsafe Buffer Copy Buffer Overflow and RAD

Experimental Computer Systems Lab Return Address Defense (RAD) Ret addr corrupted Stack Ret Addr Copy Return Address Repository Epilogue Return Address Check Buffer Overflow and RAD

Experimental Computer Systems Lab Return Address Defense (RAD) Ret addr corrupted Stack Ret Addr Copy Return Address Repository Attack !! Buffer Overflow and RAD

Experimental Computer Systems Lab Static BT & RAD – Central Issues Disassembly Code Instrumentation Static BT and RAD Binary File DisassemblerInstrumentation

Experimental Computer Systems Lab Disassembly Core component for static analysis of binaries Principal Approaches Linear Sweep Recursive Traversal Static BT and RAD

Experimental Computer Systems Lab Disassembly Core component for static analysis of binaries Principal Approaches Linear Sweep Recursive Traversal First Byte valid inst invalid inst Static BT and RAD

Experimental Computer Systems Lab Disassembly Core component for static analysis of binaries Principal Approaches Linear Sweep Recursive Traversal Entry Point CALL fn fn RET Static BT and RAD

Experimental Computer Systems Lab Disassembly Core component for static analysis of binaries Principal Approaches Linear Sweep Recursive Traversal RET Entry Point CALL fn fn RET Static BT and RAD

Experimental Computer Systems Lab Disassembly Core component for static analysis of binaries Principal Approaches Recursive Traversal Linear Sweep RET Entry Point CALL fn fn RET Static BT and RAD

Experimental Computer Systems Lab Disassembly – Impediments Code/Data distinction Variable x86 instruction size Indirect Branches Functions without explicit CALL PIC Static BT and RAD

Experimental Computer Systems Lab Disassembly – Impediments Code/Data distinction Variable x86 instruction size Indirect Branches Functions without explicit CALL PIC 0x0F 0x85 0xC0 0x0F 0x85 …. JNE offset 0x0F // data TEST eax, eax JNE offset 0x0F 0x85 0xC0 0x0F 0x850x0F0x0F 0x85 ….. Static BT and RAD

Experimental Computer Systems Lab Disassembly – Impediments Code/Data distinction Variable x86 instruction size Indirect Branches Functions without explicit CALL PIC 0x0F 0x85 0xC0 0x0F 0x85 …. JNE offset 0x0F // data TEST eax, eax JNE offset 0x0F 0x85 0xC0 0x0F 0x850x0F0x0F 0x85 ….. Static BT and RAD

Experimental Computer Systems Lab Disassembly – Impediments Code/Data distinction Variable x86 instruction size Indirect Branches Functions without explicit CALL PIC Static BT and RAD

Experimental Computer Systems Lab Disassembler – Approach Recursive Traversal Accurately code/data identification Linear Sweep Reach areas not covered by recursive traversal Compiler-independent heuristics Recover from errors Static BT and RAD

Experimental Computer Systems Lab Disassembly Accuracy Static BT and RAD

Experimental Computer Systems Lab Code Instrumentation Add desired functionality Preserve original program semantics Static BT and RAD

Experimental Computer Systems Lab Code Instrumentation – RAD Add buffer overflow protection Preserve original program semantics Static BT and RAD

Experimental Computer Systems Lab What to instrument ? Function boundary identification Prologue and epilogue must be both instrumented/uninstrumented False alarms Pattern matching (for ‘interesting’ functions) Stack frame allocation and deallocation Static BT and RAD

Experimental Computer Systems Lab What to instrument ? Binary File Disassembler Core Binary Rewrite Engine RAD Each Instruction If ‘interesting’ function Prototype

Experimental Computer Systems Lab Inserting Checking Code JMP at prologue and epilogue to RAD code Replace 5 byte worth instructions Should not disturb branch targets Stack frame allocation (at prologue) >= 5 bytes Stack frame deallocation possible in 2 – 4 bytes Return address check in INT 3 handler Static BT and RAD Prologue Epilogue JMP (save ret addr) JMP (check ret addr)

Experimental Computer Systems Lab Inserting Checking Code JMP at prologue and epilogue to RAD code Replace 5 byte worth instructions Should not disturb branch targets Stack frame allocation (at prologue) >= 5 bytes Stack frame deallocation possible in 2 – 4 bytes Return address check in INT 3 handler PUSH EBP // 1 byte MOV EBP, ESP // 2 byte SUB ESP, x // 3-6 bytes ADD ESP, x // 3-6 bytes POP EBP // 1 byte RET // 1 byte MOV ESP, EBP // 2 bytes POP EBP // 1 byte RET // 1 byte LEAVE // 1 byte RET // 1 byte INT 3 // 1 byte Static BT and RAD

Experimental Computer Systems Lab INT 3 Statistics Static BT and RAD

Experimental Computer Systems Lab Experimental Evaluation Goals of experiments Effect on program correctness Resilience to buffer overflow attacks Performance and space overhead Experimental Evaluation

Experimental Computer Systems Lab Space Overhead Micro-Benchmark Macro-Benchmark Experimental Evaluation

Experimental Computer Systems Lab Execution Time Overhead Overhead = Execution Time with RAD – Execution Time without RAD Execution Time without RAD Experimental Evaluation

Experimental Computer Systems Lab Resilience to Buffer Overflow Attack Windows Help (Winhlp32.exe) Windows NT 4.0 with Service Pack 4.0 Content file (.CNT) with long heading string Published exploit code Resists Attack !! Experimental Evaluation

Experimental Computer Systems Lab Known Limitations Disassembly Limitations Hand-crafted Assembly RAD Limitations Multi-Threaded Applications Self-Modifying Code

Experimental Computer Systems Lab Known Limitations Disassembly Limitations Indirect branches Hand-crafted Assembly RAD Limitations Multi-Threaded Applications Self-Modifying Code

Experimental Computer Systems Lab Indirect Branches and Control Flow Analysis

Experimental Computer Systems Lab Known Limitations Disassembly Limitations Hand-crafted Assembly Inter-procedural jumps, multiple function entry points RAD Limitations Multi-Threaded Applications Self-Modifying Code

Experimental Computer Systems Lab Known Limitations Disassembly Limitations Hand-crafted Assembly RAD Limitations Memory Pointer corruption Multi-Threaded Applications Self-Modifying Code

Experimental Computer Systems Lab Known Limitations Disassembly Limitations Hand-crafted Assembly RAD Limitations Multi-Threaded Applications Per-thread RAR needed Self-Modifying Code

Experimental Computer Systems Lab Known Limitations Disassembly Limitations Hand-crafted Assembly RAD Limitations Multi-Threaded Applications Self-Modifying Code Beyond static analysis

Experimental Computer Systems Lab Conclusions First study to implement Static BT state-of-the-art into a working system High disassembly accuracy Demonstrated effectiveness on commercial grade Windows applications Qualified success as a security tool Basis for a general Win32/PE Instrumentation framework

Experimental Computer Systems Lab Looking Ahead Support for DLLs Dynamic translation to ameliorate static BT deficiencies Automatic application-specific security policy generation Copy and Tamper resistant software

Experimental Computer Systems Lab Project Page Disassembler Sources

Experimental Computer Systems Lab Questions ??

Experimental Computer Systems Lab … and I’m looking for work !!!

Experimental Computer Systems Lab Disassembler – Passes DATA : : CALL 0x40011C JMP 0x : RET MOV EAX, 0x40012E; PUSH 0x CALL EAX RET MOV EAX, ECX CALL 0X x x x40011C RET 0x40012E Static BT and RAD

Experimental Computer Systems Lab

Disassembler – Passes DATA : : CALL 0x40011C JMP 0x : RET MOV EAX, 0x40012E; PUSH 0x CALL EAX RET MOV EAX, ECX CALL 0X x x x40011C RET 0x40012E Pass 1 Recursive Traversal Entry Point Inter-Procedural Call Graph Static BT and RAD

Experimental Computer Systems Lab Disassembler – Passes DATA : : CALL 0x40011C JMP 0x : RET MOV EAX, 0x40012E; PUSH 0x CALL EAX RET MOV EAX, ECX CALL 0X x x x40011C RET 0x40012E Pass 1 Recursive Traversal Entry Point Inter-Procedural Call Graph Function Body Function Control Flow Graph Backward Edge Static BT and RAD

Experimental Computer Systems Lab Disassembler – Passes DATA : : CALL 0x40011C JMP 0x : RET MOV EAX, 0x40012E; PUSH 0x CALL EAX RET MOV EAX, ECX CALL 0X x x x40011C RET 0x40012E Pass 1 Recursive Traversal Entry Point Function Body Static BT and RAD

Experimental Computer Systems Lab DATA : : CALL 0x40011C JMP 0x : RET MOV EAX, 0x40012E; PUSH 0x CALL EAX RET MOV EAX, ECX CALL 0X x x x40011C RET 0x40012E Pass 2 Linear Sweep Function Body Legal inst. Bytes as code Disassembler – Passes Static BT and RAD

Experimental Computer Systems Lab Disassembler – Passes DATA : : CALL 0x40011C JMP 0x : RET MOV EAX, 0x40012E; PUSH 0x CALL EAX RET MOV EAX, ECX CALL 0X x x x40011C RET 0x40012E Pass 3 Error Recovery Heuristics Function Body Code sequence not ending with JMP/RET => not code Static BT and RAD

Experimental Computer Systems Lab Execution Time Micro-benchmark Experimental Evaluation Penalty = Additional RAD Overhead Original Run Time

Experimental Computer Systems Lab WattProbe Linux-based experimental platform for low- power computing Power vs. Performance measurements Fine Grained High Resolution Across Multiple Resources Power characterization of interactive applications Application-specific low power modes

Experimental Computer Systems Lab Caching Infrastructure for Stackable File Systems (FAST ’03 WIP) Cache coherence support Transparent to native and stackable file systems Minimal changes to the page cache code Hooks into a cache manager Prototype Linux implementation

Experimental Computer Systems Lab Communication and Membership Management for Linux Clusters Low-latency link-level semi-reliable transport layer (Springer LNCS, HiPC ’02) No hardware support Membership Management Cluster-wide consistent membership view Scalable failure detection

Experimental Computer Systems Lab Thanks

Experimental Computer Systems Lab

Disassembly Limitations False Negatives (Functions Missed) False Positives (Falsely Identified Functions) Prototype

Experimental Computer Systems Lab Disassembly Limitations False Negatives (Functions Missed) Partly/fully misidentified as data Identified fully as code False Positives (Falsely Identified Functions Prologue Epilogue Data Function Body Uninstrumented Prologue Identified as Data Instrumented Security HoleRAR Overflow Prototype

Experimental Computer Systems Lab Disassembly Limitations False Negatives (Functions Missed) Partly/fully misidentified as data Identified fully as code False Positives (Falsely Identified Functions Prologue Epilogue Identified as Data Fn1 Fn2 Instrumented RAR overflow Uninstrumented Security Hole Prototype

Experimental Computer Systems Lab Disassembly Limitations False Negatives (Functions Missed) Partly/fully misidentified as data (pop up) Identified fully as code False Positives (Falsely Identified Functions) Data Interesting Prolog Function Body Identified as Code Security Hole Prototype

Experimental Computer Systems Lab Disassembly Limitations False Negatives (Functions Missed) False Positives (Falsely Identified Functions) Code following data/unconditional branch Missed attacks RAR overflow Indirect Jump Data Jump TargetFunction Entry Pt Prototype

Experimental Computer Systems Lab Indirect Branches and Control Flow Analysis Prototype

Experimental Computer Systems Lab RAD Limitations Attacks resisted Return Address Frame Pointer Does not protect Memory pointer attacks Function pointer corruption Import table overwrites Prototype

Experimental Computer Systems Lab Multi-Threaded Applications RAR shared between threads Possible Solution Thread Information Block (TIB) access using FS register Thread Local Storage (TLS) slots in TIB Separate RAR for each thread Thread-specific RAR addresses in TLS slots Prototype

Experimental Computer Systems Lab Hand-Crafted Assembly Quirks No interesting prolog No interesting epilog Interesting prolog Interesting epilog Fn 1 Fn 2 JNE label label Inter-procedural jump in MSAccess Fn1 entered Return address not saved Exit from Fn2 Return address check made False Alarm Prototype

Experimental Computer Systems Lab Software Architecture & Flow of Control Binary File Disassembler Core Binary Rewrite Engine RAD PE-specific component Initialization New PE Section Set up RAD mine zones Prototype Instrumentation