1. Model-Checking In-lined Reference Monitors Meera Sridhar and Kevin W. Hamlen The University of Texas at Dallas January 17, 2010 Supported in part by grants from AFOSR VMCAI 2010 Madrid, Spain

2. Reference Monitors 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 2 OS/VM R EFERENCE M ONITOR grant/denyevent Examples: file system permissions memory safety Disadvantages: changing the policy requires changing the OS/VM difficult to enforce finer-grained policies such as “No modifications to files ending in.exe” UNTRUSTED CODE

3. In-lined Reference Monitors [Schneider] 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 3 OS/VM R EFERENCE M ONITOR grant/denyevent UNTRUSTED CODE  enforce safety policies by injecting runtime security guards directly into untrusted binaries o guards test whether the impending operation constitutes a policy violation, and if so some corrective action is taken  maintain history of security-relevant events  Advantages: o No need to modify the OS/VM o enforce richer policies: e.g., no network sends after file reads o more flexible: code recipient can specify security policy  Examples: SASI [Erlingsson, Schneider], Java-MAC [Kim et al], Java-MOP [Chen, Rosu], Polymer [Bauer, Ligatti, Walker], ConSpec [Aktug, Naliuka], MoBILe [Hamlen, Morrisett, Schneider]

4. IRM Certification Problem 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 4 R EWRITER UNTRUSTED CODE POLICY VERIFIER reject (rewriter failure) execute REWRITTEN CODE Main Points: (a)TCB reduction (b)verifier must be light-weight (e.g. suitable for hand-held devices) Related work: ConSpec [Aktug, Naliuka] (certification via contracts), MoBILe [Hamlen et al] (certification via type- checking) Trusted Computing Base

5. Overview of Contributions design and implementation of prototype IRM model-checking framework for ActionScript bytecode – ActionScript/AIR ubiquitous, existing security mechanisms center around code-signing/sandboxing which are inadequate novel approach for constructing a state abstraction lattice from a security automaton – precise yet tractable abstract interpretation of IRM code rigorous proofs of soundness and convergence feasibility shown by enforcing a URL anti-redirection policy for ActionScript bytecode programs 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 5

6. ActionScript IRM Example public class RUnsafeURL extends Sprite { // constructor for popup link public function RUnsafeURL(){ var theField:TextField = new TextField(); theField.text = "Click me!"; addChild(theField); theField.addEventListener(MouseEvent.CLICK, clickListener); } // respond to user clicks by popping up infinite numbers of windows private function clickListener(e:MouseEvent):void { var url:String="javascript:window.open('http://popup.url.com');void(0);"; while(true) { navigateToURL(new URLRequest("http://popup.url.com")); } 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 6

7. ActionScript IRM Example public class RUnsafeURL extends Sprite { private var security:Number = 0; // constructor for popup link public function RUnsafeURL(){ var theField:TextField = new TextField(); theField.text = "Click me!"; addChild(theField); theField.addEventListener(MouseEvent.CLICK, clickListener); } // respond to user clicks by popping up infinite numbers of windows private function clickListener(e:MouseEvent):void { var url:String="javascript:window.open('http://popup.url.com');void(0);"; while(true) { if (security > 3) HALT; navigateToURL(new URLRequest("http://popup.url.com")); security++; } 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 7 Policy: At most 3 pop-ups

8. Reified Security State added by IRM (not in original program) abstract the event history at some point in program’s execution – Example: event counter two operations: set and test – both operations are statically identifiable – write-safety achieved via type-safety and object encapsulation (e.g., private fields) – verified by standard ActionScript bytecode verifier (similar to Java) – dangerous operations conservatively rejected (e.g., untrackable aliasing) values not necessarily current – stored state may reflect a past security state – Example: exhibit event, then increment counter later 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 8

9. Synchronizing Reified State Variables 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 9 TRACE B EFORE REIFIED STATE B EFORE C ODE L INE τ = εs = 0event e τ = es = 0s := 1 τ = es = 1event e τ = ees = 1if s =1 then … else … in practice, reified state value temporarily deviates from actual trace unavoidable in practice, must be supported Solution: represent abstract state as a pair: (S, τ ‘), abstract state when s was last updated literal event sequence subsequent to last s update (usually ε or some very short sequence)

10. Verifying IRMs with Model-Checking introduce concrete trace meta-variable τ – tracks history of security-relevant events – implicitly modified by security-relevant operations verification strategy: abstract interpret τ- enhanced program to statically verify τ ⊆ P – deciding τ ⊆ P is the model-checking step – implemented as simple regular language subset in our prototype, but more sophisticated models possible main challenge: abstracting τ – too concrete: too slow to converge – too abstract: too much conservative rejection 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 10

11. Abstract State Lattice 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 11 q1q1 q2q2 q0q0 q3q3 e4e4 e0e0 e2e2 e1e1 e3e3 T Res(q 0 ) Res(q 1 ) … …...... Res({q 0,q 1 }) SECURITY AUTOMATON STATE LATTICE ordered by subset relation we assume automata are deterministic

12. Security Policy Definition 1 (Security Policy) Let A = (Q,Σ, q 0, δ) be a deterministic security automaton. The security policy P A for automaton A is P A = Res A (Q) Res A (Q) = ∪ q ∈ Q Res A (q) where Res A (q) is the residual of state q in A. 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 12

13. Abstract Traces Definition 2 (Abstract Traces): The language SS of abstract traces is SS = {(Res(Q 0 ), τ) | Q 0 ⊆ Q, τ ∈ Σ ∗, |τ| ≤ k} ∪ { ⊤ SS } where ⊤ SS = Σ ∗. Abstract traces are ordered by subset relation ⊆, forming the lattice (SS, ⊆ ). 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 13

14. A Simple Example 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 14 Security Policy : ε + e

15. A Simple Example, contd. 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 15 Security Policy : ε + e

16. Abstract Machine 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 16 L ANGUAGE S YNTAX A BSTRACT CONFIGURATIONS

17. Abstract Small-step Operational Semantics 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 17

18. Soundness & Convergence Proof of Soundness – concrete machine defines semantics of ActionScript VM security violations modeled as stuck states – abstract machine defines semantics of abstract interpreter rejection modeled as stuck state – Progress Lemma: if the abstract machine makes progress, so does the concrete machine – Preservation Lemma: whenever both machines make progress, the abstract machine abstracts the concrete machine Proof of Convergence – abstract machine reaches a fixed point within O(n 2 ) iterations in the worst case, where n is the size of the security automaton – proof bounds the height of the abstract state lattice 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 18

19. Example Security Policy 1/17/201019 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors Policy Goal: prohibit malicious URL-redirections by ABC ad applets URL navigation in ActionScript Bytecode: ―redirections are implemented at the bytecode level by navigateToURL system calls ―our policy requires that method check_url(s) be called to validate s before any redirection to s ―check_url has a trusted implementation provided by ad distributor and/or web host, and may incorporate dynamic information) such as ad hit counts or webpage context.

20. Example Security Policy (contd.) 1/17/201020 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors Policy Enforcement: ―our IRM enforces this policy by injecting calls to check_url into untrusted applets ―for better runtime efficiency, it positions some of these calls early in the program's execution (to pre-validate certain URL's) and injects runtime security state variables that avoid potentially expensive duplicate calls by tracking the history of past calls

21. Example Security Policy (contd.) we used Prolog to build our IRM system – declarative nature of Prolog predicates yields clean, concise code – reversible nature of Prolog predicates helps reuse parsing code for code-generation some preliminary numbers: 1/17/2010 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors 21 P ROGRAM T ESTED S IZE B EFORE S IZE A FTER R EWRITING T IME V ERIFICATION T IME countdownBadge.abc 1.80 KB1.95 KB1.429s0.532s NavToURL.abc 0.93 KB1.03 KB0.863s0.233s

22. Future Work extend work to support – reified security state per-object – reified security state updated by IRM before actual security state changes at runtime support for recursion and concurrency extend work to other binary languages and IRM systems that have been implemented for them 1/17/201022 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors

23. Selected Citations 1.P. Cousot and R. Cousot. Abstract Interpretation: A unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Proc. Symposium on Principles of Prog. Languages, pages 234–252, 1977. 2.P. Cousot and R. Cousot. Abstract Interpretation Frameworks. J. Log. Comput., 2(4):511–547, 1992. 3.B. W. DeVries, G. Gupta, K. W. Hamlen, S. Moore, and M. Sridhar. ActionScript Bytecode Verification with Co-logic Programming. In Proc. of the ACM SIGPLAN Workshop on Prog. Languages and Analysis for Security (PLAS), 2009. 4.K. W. Hamlen, G. Morrisett, and F. B. Schneider. Certified In-lined Reference Monitoring on.NET. In Proc. ACM Workshop on Prog. Languages and Analysis for Security (PLAS), 2006. 5.K. W. Hamlen, G. Morrisett, and F. B. Schneider. Computability Classes for Enforcement Mechanisms. In ACM Trans. Prog. Languages and Systems, 2006. 6.F. B. Schneider. Enforceable Security Policies. ACM Trans. Information and System Security, 3:30–50, 2000. 7.M. Sridhar and K.W. Hamlen. ActionScript In-Lined Reference Monitoring in Prolog. In Proc. Intl. Symposium on Practical Aspects of Declarative Languages, 2010. 1/17/201023 Sridhar and Hamlen: Model-Checking In- lined Reference Monitors