1. © 2001 Barton P. MillerDecember 2001DynInst Security Playing Inside the Blackbox: Using Dynamic Instrumentation to Create Security Holes Barton P. Miller bart@cs.wisc.edu Computer Sciences Department University of Wisconsin Madison, Wisconsin 53705 USA

2. – 2 –© 2001 Barton P. Miller DynInst Security Overview 1. How to easily do dangerous and malicious things to a running program. 2. How to detect when someone does something evil to your program.

3. – 3 –© 2001 Barton P. Miller DynInst Security A New View Running programs are objects to be easily manipulated. Kinds of manipulations might include: qInstrumentation qOptimization qControl

4. – 4 –© 2001 Barton P. Miller DynInst Security The Vehicle:The DynInst API A machine-independent library for machine level code patching. qEases the task of building new tools. qProvides the basic abstractions to patch code on- the-fly

5. – 5 –© 2001 Barton P. Miller DynInst Security Dynamic Instrumentation qDoes not require recompiling or relinking Saves time: compile and link times are significant in real systems. Can instrument without the source code (e.g., proprietary libraries). Can instrument without linking (relinking is not always possible. qInstrument optimized code.

6. – 6 –© 2001 Barton P. Miller DynInst Security Dynamic Instrumentation (con’d) qOnly instrument what you need, when you need No hidden cost of latent instrumentation. Enables “one pass” tools. qCan instrument running programs (such as Web or database servers) Production systems. Embedded systems. Systems with complex start-up procedures.

7. – 7 –© 2001 Barton P. Miller DynInst Security The Basic Mechanism Application Program Function foo Trampoline Pre-Instrumentation Relocated Instruction Post-Instrumentation

8. – 8 –© 2001 Barton P. Miller DynInst Security The DynInst Interface qMachine independent representation qObject-based interface to build Abstract Syntax Trees (AST’s) qWrite-once, instrument-many (portable) qHides most of the complexity in the API Process Hijacker: only 700 lines of user code! MPI tracer: 250 lines

9. – 9 –© 2001 Barton P. Miller DynInst Security Basic DynInst Operations qProcess control: Attach/create process Monitor process status changes Callbacks for fork/exec/exit qImage (executable program) routines: Find procedures/modules/variables Call graph (parent/child) queries Intra-procedural control-flow graph

10. – 10 –© 2001 Barton P. Miller DynInst Security Basic DynInst Operations qInferior (application processor) operations: Malloc/free –Allocate heap space in application process Inferior RPC –Asynchronously execute a function in the application. Load module –Cause a new.so/.dll to be loaded into the application.

11. – 11 –© 2001 Barton P. Miller DynInst Security Basic DynInst Operations qInferior operations (continued): Remove Function Call –Disable an existing function call in the application Replace Function Call –Redirect a function call to a new function Replace Function –Redirect all calls (current and future) to a function to a new function.

12. – 12 –© 2001 Barton P. Miller DynInst Security Basic DynInst Operations qBuilding AST code sequences: Control structures: if and goto Arithmetic and Boolean expressions Get PID/TID operations Read/write registers and global variables Read/write parameters and return value Function call

13. – 13 –© 2001 Barton P. Miller DynInst Security Security Applications of DynInst Lots of tool applications of Dyninst by lots of groups. Here are two security-oriented ones: qLicense server bypassing qCondor security attacks

14. – 14 –© 2001 Barton P. Miller DynInst Security License Server Attack: The Bypass Program License Data Network License Server Normal: licensed program runs after communicates with license server. Program License Data Network License Server Undesired: licensed program refuses to run if license server does not respond.

15. – 15 –© 2001 Barton P. Miller DynInst Security Example: Adobe FrameMaker Two-step license verification: retrieve license data from server [once] check license data for correctness [often] In practice: allow FM to time-out waiting for server allow FM to attempt to go into “demo” mode switch FM back to normal mode insure that future license checks always succeed

16. – 16 –© 2001 Barton P. Miller DynInst Security Strategies qComplete reverse engineering: not an option –legal problems –complexity (FrameMaker is a 7 MB binary!) qFocus on certain characteristics: I/O (network sockets) traffic execution trace

17. – 17 –© 2001 Barton P. Miller DynInst Security Tools qHigh-level language translators: Dyner: interactive, interpreted C subset Jdyninst: Java to DynInst compiler qBypasser: an interactive call graph browser Search and walk application call graph Resolves function pointers at runtime Call follow caller or callee paths Can generate call trace

18. – 18 –© 2001 Barton P. Miller DynInst Security Use qDetermining where to apply changes: get trace for a successful run get trace for a (forced-)failure run compare to find differences repeat as needed

19. – 19 –© 2001 Barton P. Miller DynInst Security Details  FM calls NlOpenlicenses on start up Contacts license server and caches credential if successful  At end of main, FM calls NluiCheckLicense If credential is not present, call ChangeProductToDemo (cannot save files) qFrequently, during operation, FM will check for cached credentials.

20. – 20 –© 2001 Barton P. Miller DynInst Security Details  FM calls NlOpenlicenses on start up Contacts license server and caches credential if successful Allow this to fail.  At end of main, FM calls NluiCheckLicense If credential is not present, call ChangeProductToDemo (cannot save files) qFrequently, during operation, FM will check for cached credentials.

21. – 21 –© 2001 Barton P. Miller DynInst Security Details  FM calls NlOpenlicenses on start up Contacts license server and caches credential if successful Allow this to fail.  At end of main, FM calls NluiCheckLicense If credential is not present, call ChangeProductToDemo (cannot save files) Delete the call to ChangeProductToDemo. qFrequently, during operation, FM will check for cached credentials.

22. – 22 –© 2001 Barton P. Miller DynInst Security Details  FM calls NlOpenlicenses on start up Contacts license server and caches credential if successful Allow this to fail.  At end of main, FM calls NluiCheckLicense If credential is not present, call ChangeProductToDemo (cannot save files) Delete the call to ChangeProductToDemo. qFrequently, during operation, FM will check for cached credentials. Change this call to always return “true”.

23. – 23 –© 2001 Barton P. Miller DynInst Security Condor Attack: Lurking Jobs qCondor schedules jobs on idle workstations qIn a normal mode, jobs run as a common, low- privilege user ID: “nobody”. qThis common user ID provides an opportunity for an evil lurking process to ambush subsequent jobs (from other users):

24. – 24 –© 2001 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host User Job system calls

25. – 25 –© 2001 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host Evil User Job system calls Lurker Process fork

26. – 26 –© 2001 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Execution Host Lurker Process

27. – 27 –© 2001 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host Innocent User Job system calls Lurker Process

28. – 28 –© 2001 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host Innocent User Job system calls Lurker Process attach

29. – 29 –© 2001 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host Innocent User Job system calls Control remote system calls Lurker Process rm -rf *

30. – 30 –© 2001 Barton P. Miller DynInst Security

31. – 31 –© 2001 Barton P. Miller DynInst Security

32. – 32 –© 2001 Barton P. Miller DynInst Security Can We Trust a Remote Job? The threats: 1. Cause the job to make improper remote system calls. 2. Cause the job to calculate an incorrect answer. 3. Steal data from the remote job. Threat protection strategies: File sand-boxing (#1) System call sand-boxing (#1) Obscure and encode binary (#1) Replicate remote job (#2)

33. – 33 –© 2001 Barton P. Miller DynInst Security Sand-Boxing Shadow process selectively rejects system calls: Restrict access to specific files or directories Disallow certain system calls Disallow certain system call parameter values Submitting Host Shadow Process Execution Host User Job system calls

34. – 34 –© 2001 Barton P. Miller DynInst Security Obscuring the Executable User Job Modified User Job Checking Shadow Modifier/Obscurer

35. – 35 –© 2001 Barton P. Miller DynInst Security Obscuring the Executable Goal: Even if an intruder can see, examine, and fully control the remote job, no harm can come to the local machine.

36. – 36 –© 2001 Barton P. Miller DynInst Security Obscuring the Executable Modify the executable: Replace each system call site with call to a unique random function name This makes understanding the executable harder and makes checking the system call stream easier. Delete any constant parameters Values like file names or descriptor numbers are hidden. Pad parameter other lists so all calls have equal number Permute the order of parameters Add dummy system calls throughout code

37. – 37 –© 2001 Barton P. Miller DynInst Security Obscuring the Executable Generate a Checking Shadow: Build a FSM that reflects the inter-procedural control-flow of the job. Note that this FSM has increased accuracy, since each call site has a unique name Map random function names back to their actual system calls. Insert parameter values and present in correct order. Ignore dummy calls.

38. – 38 –© 2001 Barton P. Miller DynInst Security Obscuring the Executable Some notes of interest: All hiding and replacement is done before the job is submitted to Condor. No knowledge of the encryption or keys is contained in the remote job. All replacing is done in the Checking Shadow, which only resides on your local machine. FSM accuracy and computational complexity is simplified by the function renaming. Our early results look good. A tech report will be available by the end of November.

39. – 39 –© 2001 Barton P. Miller DynInst Security How to Get a Copy of DynInst: Release 2.3 (release 3.0 imminent) Free for research use. Runs on Solaris (SPARC & x86), Windows NT, AIX/SP2, Linux (x86), Irix (MIPS),Tru64 Unix (Alpha). http://www.paradyn.org http://www.dyninst.org paradyn@cs.wisc.edu