1. Program Obfuscation: A Quantitative Approach Presented by: Mariusz Jakubowski Microsoft Research Third Workshop on Quality of Protection October 29 th, 2007 Bertrand Anckaert, Matias Madou, Bjorn De Sutter, Bruno De Bus, Koen De Bosschere, and Bart Preneel Ghent University and K.U.Leuven, Belgium

2. Obfuscation has many applications 2

3. There is a large gap between theoretical results 3 - On the (Im)possibility of Obfuscating Programs – Barak et al. (2001) - On the Impossibility of Obfuscation with Auxiliary Input – Goldwasser et al. (2005) - Positive Results and Techniques for Obfuscation – Lynn et al. (2004) - Towards Realizing Random Oracles: Hash Functions that Hide All Partial Information Canetti et al. (1997) + - Large gap Intuitively, obfuscation does help

4. We need a practical system for evaluating obfuscating transformations It should be easy to evaluate existing and future transformations => Automated The evaluation should convey difficulty of reverse-engineering => Build upon experience from complexity metrics 4

5. oIntro oMetrics oInstruction Count oCyclomatic Number oKnot Count o(De)Obfuscating transformations Outline 5

6. Four axes based on typical reverse- engineering scenario 6 Disassemble Flow graph construction Analyse Data Flow Interpret Data Code Control flow Data flow Data

7. + No uncertainty about executed code + Always availabe - Only about covered part of the code Evaluated Complexity Metrics 7 Code Control flow Data flow Data Instruction Count Cyclomatic Number Knot Count Metrics are collected by a run-time instrumentation framework

8. Cyclomatic number and knot count Cyclomatic number: – #edges – #nodes + 2 – Intuitively: the number of decision points Knot count: – #crossings – Intuitively: the unstructuredness 8

9. oIntro oMetrics o (De)Obfuscating transformations oJump redirection [Linn et al. 2003] oControl flow flattening [Chenxi Wang et al. 2001] oOpaque predicates [Collberg et al. 1998] Outline 9

10. Jump redirection Redirect branches to function 10 1 Jmp 2 2 1 call branch Branch Function 2 garbage assumed return site

11. Impact of Jump Redirection 11

12. Jump redirection - deobfuscation Identify Branch Function – signature based – run-time behavior Record (call,return) pairs under debugger Overwrite calls 12 1 call branch Branch Function 2 garbage assumed return site (1,2) (4,7) (9,5) … jmp 2

13. Success of De-obfuscation 13

14. Control flow flattening All original basic blocks have the same predecessor and successor 14 1 4 32 1 43 2 switch

15. Control flow flattening significantly increases the complexity metrics 15

16. Success of De-obfuscation 16

17. Opaque predicates 17 1 Jmp 2 2 1 Jmp if (2==2) 2fake Add fake decision statements

18. Impact of Opaque Predication 18

19. Conclusion A first step towards a unified quantitative evaluation of – obfuscating transformations – deobfuscating transformations Which leverages experience from the established field of complexity metrics 19

20. Program Obfuscation: A Quantitative Approach Presented by: Mariusz Jakubowski Microsoft Research Third Workshop on Quality of Protection October 29 th, 2007 Bertrand Anckaert, Matias Madou, Bjorn De Sutter, Bruno De Bus, Koen De Bosschere, and Bart Preneel Ghent University and K.U.Leuven, Belgium