Environment Selection Application  Firefox 1.0 or 2.0  Apache Operating System  Linux  Windows XP Instrumentation Package  JIT (DynamoRio, Pin)  Trampoline (Dyninst, Detours, Pin, etc)

Application Firefox 1.0 (Phase 1)  Complex app with embedded interpreter  39 to 46 applicable vulnerabilities Firefox 2.0  Similar vulnerabilities as 1.0 Apache  Less complex application  6-8 applicable vulnerabilities Proposal: Firefox 1.0  Many interesting vulnerabilities  Leverages Phase 1 experience

Operating System Linux  Open source  Open source tools (gcc, Xnee, etc)  Instrumentation tools are supported better Windows XP  Closed source  More marketable results Proposal: Windows XP  No show stoppers for Windows  Shows program is more generally applicable

Instrumentation Tools Instrumentation tool approaches  JIT  Probe based Call interception  System call  Library call

JIT Binary Translation PIN & DynamoRIO Allows us, at runtime, to manipulate every instruction, with:  Minimal performance overhead  Full transparency Exports interface for building custom tools No modifications to hardware, operating system, or application

How does it work? (conceptually) fetchdecodeexecute Start

In more detail 120% to 200%

JIT-mode Summary Powerful instruction-level instrumentation  Supports shadow stack  Supports arbitrary repairs  Stack-walk Direct access to system call gateway

Probe based instrumentation: PIN probe, Dyninst, Detours

Probe-based Repair

Probe-mode Summary Considerably faster than JIT-mode  No constant performance overhead Potential issues  x86: need at least 5 bytes for trampolines  Can be expensive for fine-grained instrumentation  Limited to function-level instrumentation  Does not support shadow stack

Direct System Call Interception Application System call gateway Operating System Interception

Library System Call Interception Application System call gateway Operating System Win32 API Win32 DLLs Interception

Issue With Library Interception Can only catch system calls made through API (libc, win32API) Malicious attacker could inject a different version of the library we are intercepting  But that would require code-injection

Library Interception Can only catch system calls made through API (libc, win32API) Malicious attacker could inject a different version of the library we are intercepting  But that would require code-injection Stable, coherent interface

Monitor/Repair Matrix ToolTypeOSStack Replace Args Change or drop syscall Syscall return value Perform ance PINJIT Win, Linux SS SW YYY 500% 240% PINProbe Win, Linux SWLLL180% DRJIT Win, Linux SS SW YLL 400% 220% DetoursProbeWinSWLLL~180% DyninstProbeLinuxSWLLL~180%

PIN Automatically in-lines instrumentation code  Uses callouts ‣ More expensive but easy to write  No restrictions on library usage Simple, easy-to-use API Works on Linux and Windows Two modes of operation: JIT and Probe  Cover both models we want to use Only slightly slower than DynamoRIO

DynamoRIO Lower level interface Library calls are constrained  Must use DR version of calls (e.g., malloc)  Some calls (e.g., sockets) not supported Does not allow direct manipulation of system calls Just released as open source Phase 1 code (shadow stack, HeapGuard) now available

Plan Use Pin to develop prototype  Supports both JIT and Probe  Easy to use Implement final approach later  Evaluate numerous exploits  Understand what our needs are Options  Use probe mode if possible  Consider DynamoRio if necessary for speed and/or flexibility

Conclusion Application: Firefox 1.0 Operating System: Windows XP Instrumentation: Pin for now

Windows-Linux: Development Windows is closed source  Forced to reverse engineer Windows (and its tools) to debug problems  Visual Studio compiler is closed source ‣ Difficult to debug ‣ Cygwin environment has issues

Accomplishments Monitoring framework Analysis framework Reproduction framework

Monitoring Framework PIN-based monitoring tool Two modes of operation  JIT  Probe Analysis  Shadow stack  Stack walk (after we disable FP optimization)

Analysis Framework Tools for analyzing callstack information  Finite state automata data-structure  Visual representation  Suffix-tree fast lookup comparisons Implemented in Python  Using networkx libraries  Cross platform

Reproduction Framework Automate training  Record & replay user interactions with FF ‣ Record mouse & keyboard events  Works in Linux ‣ Using Xnee

Windows - Linux Pin Performance Windows  Shadow Stack: ~4.5x  Stack Walk: ~0.40x Linux  Shadow Stack: ~4.0x  Stack Walk: ~0.30x