1. Runtime Protection via Dataflow Flattening Bertrand Anckaert Ghent University/ Boston Consulting Group The Third International Conference on Emerging Security Information, Systems and Technologies SECURWARE 2009 June 18-23, 2009 – Athens/Glyfada, Greece Mariusz H. Jakubowski Ramarathnam Venkatesan Chit Wei (Nick) Saw Microsoft Research Redmond, WA (USA)

2. SECURWARE 2009June 18-23, 20092 Introduction Software protection –Complicate reverse engineering and tampering. –Enforce execution as intended by developer. –DRM, licensing, anti-malware, OS security, etc. Dataflow analysis –Track flow of data through program. –Locate and tamper “interesting” data. Goals of our work: –Develop methods against malicious dataflow analysis. –Study dataflow flattening as an element of comprehensive protection frameworks.

3. SECURWARE 2009June 18-23, 20093 Overview Introduction Background Dataflow flattening Implementation and experiments Applications Conclusion Protecting data operations

4. SECURWARE 2009June 18-23, 20094 Background Oblivious RAMs [Goldreich and Ostrovsky ’96] –Randomized memory-access patterns –Each fetch/store replaced by many fetch/stores –Cannot infer program operation from memory accesses Control-flow flattening –Program’s CFG converted to flat (two-level) graph –Cannot infer control-flow structure from execution on the flat graph

5. SECURWARE 2009June 18-23, 20095 Overview Introduction Background Dataflow flattening Implementation and experiments Applications Conclusion Protecting data operations

6. SECURWARE 2009June 18-23, 20096 Dataflow Flattening Two main aspects: –Making dataflow graph appear complete –Randomizing memory-access patterns Informally: –“Every variable affects every other variable.” –“Program accesses memory at random.”

7. SECURWARE 2009June 18-23, 20097 Flattening Dataflow Graphs Basic idea: Make dataflow graph appear complete. –Every variable affects all other variables. –Cannot infer useful variable dependencies. Data-centric analog of control-flow flattening AB X Y

8. SECURWARE 2009June 18-23, 20098 Flattening Dataflow Graphs Basic idea: Make dataflow graph appear complete. –Every variable affects all other variables. –Cannot infer useful variable dependencies. Data-centric analog of control-flow flattening AB X Y

9. Heap Memory Management Unit Program Dataflow Flattening via an MMU

10. SECURWARE 2009June 18-23, 200910 MMU Software-based Memory Management Unit: –Periodic reordering of heap data –Migration of variables from stack to heap –Pointer masking Variable references redirected through MMU

11. SECURWARE 2009June 18-23, 200911 MMU Operation Heap subdivided into encrypted pages (e.g, 4KB). Upon each access of a heap page: –Retrieve n extra pages with probability 1/p. –Randomly shuffle the (expected) 1+n/p pages. –Re-salt and re-encrypt each page.

12. SECURWARE 2009June 18-23, 200912 Security Analysis Security analyzed via practical metrics Confusion factor C as a metric –Define C as the number of possible places for a page in memory. –Let N = total number of memory pages. –Oblivious RAMs: C = N after each memory access. –Our approach: C converges to N as accesses occur.

13. SECURWARE 2009June 18-23, 200913 Practical Issues Most program variables are stack-based. –Solution: Migrate variables from stack to heap. –Explicitly allocate and free heap memory when entering and exiting stack frames. Pointers can reveal access patterns. –Solution: Scramble pointers. –Only MMU knows mapping between addresses and variables.

14. SECURWARE 2009June 18-23, 200914 Overview Introduction Background Dataflow flattening Implementation and experiments Applications Conclusion Protecting data operations

15. SECURWARE 2009June 18-23, 200915 Implementation Tool for transforming C programs –Based on Phoenix compiler infrastructure –Compiler backend plug-in –Instrumentation of Phoenix IR Transformations –Interception and custom implementation of heap operations (malloc, free, etc.) –Conversion of stack variables to heap variables –Pointer scrambling (encryption)

16. SECURWARE 2009June 18-23, 200916 Experimental Results Selected SPEC benchmarks (compression, optimization, QCD, chess, fluid dynamics ) Simple algorithms (pseudorandom-number generation, summing a list of integers) Performance impact (slowdown)

17. SECURWARE 2009June 18-23, 200917 Overview Introduction Background Dataflow flattening Implementation and experiments Applications Conclusion Protecting data operations

18. SECURWARE 2009June 18-23, 200918 Applications Software protection –Anti-malware systems –Licensing, DRM, product activation, etc. –Defenses against information-extraction and side- channel attacks More comprehensive tools –Element of broader protection strategies –Means of realizing “engineering assumptions” needed by some security models

19. SECURWARE 2009June 18-23, 200919 Conclusion Dataflow flattening –Makes dataflow graph appear complete. –Randomizes memory-access patterns. –Complicates inference of algorithms from their data operations. Future directions –Dataflow flattening as part of more comprehensive systems –Security analysis via models and metrics