1. © 2000 Barton P. MillerSeptember 6, 2000DynInst Security Playing Inside the Blackbox: Using Dynamic Instrumentation to Create Security Holes Barton P. Miller bart@cs.wisc.edu Computer Sciences Department University of Wisconsin Madison, Wisconsin 53705 USA

2. – 2 –© 2000 Barton P. Miller DynInst Security A New View Running programs are objects to be easily manipulated. Kinds of manipulations might include: qInstrumentation qOptimization qControl qSubversion?

3. – 3 –© 2000 Barton P. Miller DynInst Security The Vehicle:The DynInst API A machine-independent library for machine level code patching. qEases the task of building new tools. qProvides the basic abstractions to patch code on- the-fly

4. – 4 –© 2000 Barton P. Miller DynInst Security Dynamic Instrumentation qDoes not require recompiling or relinking Saves time: compile and link times are significant in real systems. Can instrument without the source code (e.g., proprietary libraries). Can instrument without linking (relinking is not always possible. qInstrument optimized code.

5. – 5 –© 2000 Barton P. Miller DynInst Security Dynamic Instrumentation (con’d) qOnly instrument what you need, when you need No hidden cost of latent instrumentation. Enables “one pass” tools. qCan instrument running programs (such as Web or database servers) Production systems. Embedded systems. Systems with complex start-up procedures.

6. – 6 –© 2000 Barton P. Miller DynInst Security The Basic Mechanism Application Program Function foo Trampoline Pre-Instrumentation Relocated Instruction Post-Instrumentation

7. – 7 –© 2000 Barton P. Miller DynInst Security The DynInst Interface qMachine independent representation qObject-based interface to build Abstract Syntax Trees (AST’s) qWrite-once, instrument-many (portable) qHides most of the complexity in the API Process Hijacker: only 700 lines of user code! MPI tracer: 250 lines

8. – 8 –© 2000 Barton P. Miller DynInst Security Basic DynInst Operations qProcess control: Attach/create process Monitor process status changes Callbacks for fork/exec/exit qImage (executable program) routines: Find procedures/modules/variables Call graph (parent/child) queries

9. – 9 –© 2000 Barton P. Miller DynInst Security Basic DynInst Operations qInferior (application processor) operations: Malloc/free –Allocate heap space in application process Inferior RPC –Asynchronously execute a function in the application. Load module –Cause a new.so/.dll to be loaded into the application.

10. – 10 –© 2000 Barton P. Miller DynInst Security Basic DynInst Operations qInferior operations (continued): Remove Function Call –Disable an existing function call in the application Replace Function Call –Redirect a function call to a new function Replace Function –Redirect all calls (current and future) to a function to a new function.

11. – 11 –© 2000 Barton P. Miller DynInst Security Basic DynInst Operations qBuilding AST code sequences: Control structures: if and goto Arithmetic and Boolean expressions Get PID/TID operations Read/write registers and global variables Read/write parameters and return value Function call

12. – 12 –© 2000 Barton P. Miller DynInst Security Applications of DynInst qProcess Hijacking (Vic Zandy) Submitting already-running jobs to Condor qMPI Tracer (Chris Chambreau) Insert Vampir or Pablo trace calls on the fly. qFunction Call Tracer (Roland Wismüller) Generate dynamic call graph qImage Mentor (Brian Wylie) Query module/function/memory structure qRe-Tee (Jeff Hollingsworth) Redirect program output on-the-fly qLicense server bypassing qCondor security attacks

13. – 13 –© 2000 Barton P. Miller DynInst Security License Server Attack: The Bypass Program License Data Network License Server Normal: licensed program runs after communicates with license server. Program License Data Network License Server Undesired: licensed program refuses to run if license server does not respond.

14. – 14 –© 2000 Barton P. Miller DynInst Security Example: Adobe FrameMaker Two-step license verification: retrieve license data from server [once] check license data for correctness [often] In practice: allow FM to time-out waiting for server allow FM to attempt to go into “demo” mode switch FM back to normal mode insure that future license checks always succeed

15. – 15 –© 2000 Barton P. Miller DynInst Security Strategies qComplete reverse engineering: not an option –legal problems –complexity (FrameMaker is a 7 MB binary!) qFocus on certain characteristics: I/O (network sockets) traffic execution trace

16. – 16 –© 2000 Barton P. Miller DynInst Security Tools qHigh-level language translators: Dyner: interactive, interpreted C subset Jdyninst: Java to DynInst compiler qBypasser: an interactive call graph browser Search and walk application call graph Resolves function pointers at runtime Can follow caller or callee paths Can generate call trace

17. – 17 –© 2000 Barton P. Miller DynInst Security Use qDetermining where to apply changes: get trace for a successful run get trace for a (forced-)failure run compare to find differences repeat as needed

18. – 18 –© 2000 Barton P. Miller DynInst Security Details  FM calls NlOpenlicenses on start up Contacts license server and caches credential if successful  At end of main, calls NluiCheckLicense If credential is not present, call ChangeProductToDemo (cannot save files) qFrequently, during operation, FM will check for cached credentials.

19. – 19 –© 2000 Barton P. Miller DynInst Security Details  FM calls NlOpenlicenses on start up Contacts license server and caches credential if successful Allow this to fail.  At end of main, calls NluiCheckLicense If credential is not present, call ChangeProductToDemo (cannot save files) qFrequently, during operation, FM will check for cached credentials.

20. – 20 –© 2000 Barton P. Miller DynInst Security Details  FM calls NlOpenlicenses on start up Contacts license server and caches credential if successful Allow this to fail.  At end of main, calls NluiCheckLicense If credential is not present, call ChangeProductToDemo (cannot save files) Delete the call to ChangeProductToDemo. qFrequently, during operation, FM will check for cached credentials.

21. – 21 –© 2000 Barton P. Miller DynInst Security Details  FM calls NlOpenlicenses on start up Contacts license server and caches credential if successful Allow this to fail.  At end of main, calls NluiCheckLicense If credential is not present, call ChangeProductToDemo (cannot save files) Delete the call to ChangeProductToDemo. qFrequently, during operation, FM will check for cached credentials. Change this call to always return “true”.

22. – 22 –© 2000 Barton P. Miller DynInst Security Condor Attack: Lurking Jobs qCondor schedules jobs on idle computers. qIn a normal mode, jobs run as a common, low- privilege user ID: “nobody”. qThis common user ID provides an opportunity for an evil lurking process to ambush subsequent jobs (from other users):

23. – 23 –© 2000 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host User Job system calls

24. – 24 –© 2000 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host Evil User Job system calls Lurker Process fork

25. – 25 –© 2000 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Execution Host Lurker Process

26. – 26 –© 2000 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host Innocent User Job system calls Lurker Process

27. – 27 –© 2000 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host Innocent User Job system calls Lurker Process attach

28. – 28 –© 2000 Barton P. Miller DynInst Security Condor Job Structure Submitting Host Shadow Process Execution Host Innocent User Job system calls Control remote system calls Lurker Process rm -rf *

29. – 29 –© 2000 Barton P. Miller DynInst Security How to Get a Copy of DynInst: Release 1.3 Free for research use. Runs on Solaris (SPARC & x86), Windows NT, AIX/SP2, Linux (x86), Irix (MIPS),Tru64 Unix (Alpha). http://www.cs.wisc.edu/paradyn paradyn@cs.wisc.edu