1. Formal Refinement of Obfuscated Codes Hamidreza Ebtehaj 1

2. What is an Obfuscation Obfuscation is the act of making something obscure, perplexing or confusing in order to prevent others from understanding the obfuscated item. Let be a program transformation from a source program p into a target program p’, is an obfuscating transformation if: 1.the transformation T is potent and 2.p and p’ have the same observable behavior, 1.p fails to terminate => p′ may or may not terminate, 2.P terminates => p′ must terminate with same result as p. 2

3. Potency A program transformation is potent if the transformed (obfuscated) program is more complex to understand than the original one. Metrics of Potency : 1.The length of the program (the number of instructions and arguments) 2.The nesting level (the number of nested conditions) 3.The data flow (the number of references to local variables) 4.The data structure complexity 5.The control flow 3

4. Measures of Obfuscations Potency Cost Stealth –An obfuscating transformation is stealthy if it does not “stand out” from the rest of the program. Resilience –The resilience of a transformation measures how difficult it is to break for an automatic deobfuscator. 4

5. Taxonomy of Obfuscations There is no standard taxonomy of Obfuscations Types of obfuscation : –Anti-Static Analysis –Anti-Dynamic Analysis Types of obfuscation : –Layout obfuscators –Data obfuscators –Control code obfuscators –Opaque Predicates 5

6. Layout Obfuscators Layout obfuscating transformations act on code information that is unnecessary to its execution 6

7. Data Obfuscators Data obfuscators operate on program data structures for (int i=1; i < 1000; i++)≡ for (int i’=11; i’ < 8003; i++) {i’ = 8*i+3} Trade-off between resilience and potency on one hand, and cost on the other hand Ordering transformations 7

8. Control code obfuscators Control obfuscations attempt to confuse the program control flow –inline procedures –loop transformations: loop unrolling –Control Flow Flattening These transformations have a low resilience when applied in isolation High resilience when combined together. Control transformations often rely on the existence of opaque predicates. 8

9. Opaque Predicates Opaque predicates are often used to design control code obfuscating transformations that are cheap and resilient to attacks from deobfuscators. An opaque predicate is a predicate whose constant value is known at obfuscation time. 9

10. Some Methods 1.Self Modifying codes 2.Polymorphism & Metamorphism 3.Basic Flattening 4.Disassembly Desynchronization 5.Virtualization Obfuscation 6.Debugger & Sandbox Detection 7.API Redirection 8.Stolen Bytes & Stolen Codes 10

11. Why do we need Obfuscations? Reverse Engineering –making the task of reverse engineering the program “too expensive” in terms of the resources or time required to do so. –Hiding Algorithms Protect “software watermarks” and fingerprint Avoiding of software piracy –Cracking / Keygening Increasing Security –Hiding bugs and vulnerabilities. Hide malicious content 11

12. Basic Control Flow Flattening Control flow flattening aims to obscure the control flow logic of a program by “flattening” the control flow graph so that all basic blocks appear to have the same set of predecessors and successors. 12

13. Deobfuscation Methods Graph Reachability Algorithms: DFS/BFS –Problem: Opaque Predicates –Our control flow is more like a program graph Model Checking –Problem: State space explosion Sharath K. Udupa, Saumya K. Debray –Combining with Dynamic Analysis. –Problem : Overestimation and Underestimation errors –They can’t rebuild Original Control Flow Our Method (Inspired from Mana Taghdiri) 13

14. Our Method 14

15. Alloy Review Everything in Alloy is a relation Signatures and Atoms –Extends –sig FSObject { parent: lone Dir } –is it enough? Relational Join (. Operator) 15

16. Alloy Review Relational Product : (->) Relational Inverse : (~) Transitive Closure : (^, *) unary operator 16

17. Our Method 17

18. Abstraction Phase We Abstract each Basic Block in such a way that it’s possible to go from a basic block to any other. 18

19. Translation Phase Jalloy Local State Encoding Modeling Object References Explicitly I haven’t written the rules for translation! –We define types –Program States –Loop Unrollment 19

20. Some Definitions We use program points to denote the control points in a program. A program point corresponding to a Basic Block is called a call point. A program state σ is defined at each program point π as a mapping from variables accessible at π to some values. The set of all possible states of a program is denoted by Σ. 20

21. Some Definitions Each program statement s is viewed as a transition in the state of the program and thus, is represented by a set of pairs of program states, i.e. [s] ⊆ Σ × Σ. A pair (σ,σ’) is included in the set defining a statement s if and only if executing s in the state σ can result in the state σ’. 21

22. Abstract trace An abstract trace denotes an execution of an abstract program represented by a sequence of pairs of program points and program states : A trace t is valid if and only if : An invalidity witness is a triple of a program point and two program states (π,σ,σ’) where. (σ,σ’) / ∈ [s_π]. 22

23. Solving Phase Abstract trace : (s0,[0,0]) (s1,[0,0]) (s5,[0,0]) (s6,[0,0]) (s10,[0,0]) (s11,[0,0]) (s15,[3,3]) 23

24. Validity Check (s0,[0,0]) (s1,[0,0]) (s5,[0,0]) (s6,[0,0]) (s10,[0,0]) (s11,[0,0]) (s15,[3,3]) 24

25. Spec Inference 25

26. Problems We don’t have implemented tools Loop Unrollment can produce some errors –Solution: changing our plan, horn clauses, model checking 26

27. Happy to hear your comments سعدیا گر همه شب شرح غمش خواهی گفت /شب به پایان رود و شرح به پایان نرود 27