Enforcing Non-safety Security Policies with Program Monitors Jay Ligatti (Princeton University); joint work with: Lujo Bauer (Carnegie Mellon University), David Walker (Princeton University)

Security Policy Enforcement News flash: Software sometimes does bad stuff Bugs Malicious design One mitigation is run-time monitoring Ensure that software adheres to run-time constraints specified by a security policy Stack inspection, file access control, applet sandboxing, firewalls, resource monitors, …

Run-time Program Monitors Untrusted Target Program Monitor Executing System Open(f,“w”) is OK Open(f,“w”) Open(f,“w”) Monitors enforce policies by: Interposing between untrusted code and the system executing the untrusted code Making sure only safe code is executed

Research Agenda What are the limits of program monitors? Which policies can monitors enforce? Understanding limits helps us when selecting enforcement mechanisms Allows us to choose the simplest mechanism that enforces a policy Prevents us from attempting to enforce a policy with an insufficiently strong mechanism

Outline Motivation Defining “monitor enforces a policy” Which policies can monitors enforce? Defining “monitor enforces a policy” Define policies Define monitors Define enforcement of policies by monitors Delineating the enforceable policies Infinite renewal properties Comparison of renewal, safety, and liveness Summary

Systems and Executions System = a state machine that transitions states by executing actions We specify a system according to the possibly countably infinite set of actions it can execute A = { open f, close f } (where f is any file name) Execution = possibly infinite sequence of actions open f; close f open f1; close f1; open f2; close f2; …

Execution Notation On a system with action set A, A* = set of all finite executions Aω = set of all infinite executions A∞ = set of all executions The empty sequence (execution) is ● The concatenation of sequences s and u is s;u Prefix notation: s≤u (or u≥s) Means: s is a finite prefix of possibly infinite u Read: “s prefixes u” (or “u extends s”) We’ll need some notation for building up executions and sets of executions from actions.

Policies [Schneider ’00] Policy = predicate P on sets of executions Set of executions S Í A∞ satisfies P iff P(S) Termination policy P(S) iff every execution in S is finite Key uniformity policy P(S) iff the cryptographic keys used in all executions in S form uniform distribution

Properties [Schneider ’00] Policy P is a property iff $ predicate p on executions such that "S Í A∞ , P(S) Û "uÎS : p(u) Properties cannot specify relationships between executions Termination is a property Key uniformity is not a property Monitors enforce properties, not general policies A policy is a property iff we can decide whether some set of executions satisfies the policy just by examining every execution in the set in isolation.

Safety and Liveness [Lamport ’77; Alpern, Schneider ’85] Two types of properties have been studied a lot Safety: “Bad executions cannot be made good” "sÎA∞ : Øp(s) Þ $s’≤s : "u≥s’ : Øp(u) Access-control (cannot “undo” illegal accesses) Liveness: “Finite executions can be made good” "sÎA* : $u≥s : p(u) Termination and nontermination

Outline Motivation Defining “monitor enforces a policy” Which policies can monitors enforce? Defining “monitor enforces a policy” Define policies Define monitors Define enforcement of policies by monitors Delineating the enforceable policies Infinite renewal properties Comparison of renewal, safety, and liveness Summary

Operation of Monitors: Accepting an OK Action Untrusted Target Program Monitor Executing System Open(f,“w”) Open(f,“w”) is OK Open(f,“w”) Monitor inputs actions from target and outputs actions to the executing system Here, input action is safe to execute, so monitor accepts it (makes it observable)

Operation of Monitors: Suppressing an Action Untrusted Target Program Monitor Executing System Open(f,“w”) Open(f,“w”) is not OK Input action is not safe to execute, so monitor suppresses it and allows target to continue executing

Operation of Monitors: Inserting an Action Untrusted Target Program Monitor Executing System Open(f,“w”) Open(f,“w”) is not OK Close(f,“w”) Input action is not safe to execute, so monitor inserts another action, then reconsiders the original action

Modeling Monitors t : Q x A ® Q x (A U {●}) Model a monitor that can accept, suppress, and insert actions as an edit automaton (Q,q0,t) Q is finite or countably infinite set of states q0 is initial state A complete, deterministic, and TM-decidable function t : Q x A ® Q x (A U {●}) this transition function does not directly model the acceptance of an action. accept an action by first inserting it into the output and then suppressing it from the input current state input action new state action to insert suppress input action

Operational Semantics A monitor’s transition function defines how it behaves on individual input actions For the definition of enforcement, we need to consider the way monitors transform entire input executions Monitors are execution transformers Untrusted input Valid output a1;a2;a2;a3;… a1;a2;a2;a4;… Monitor

Operational Semantics Judgments Desired judgment: (q0,s) X ß u Automaton X starting in state q0 transforms input sequence s into output sequence u Build up to this judgment 1. Single-step judgment (q,s) X ®u (q’,s’) 2. Multi-step judgment (q,s) X Þu (q’,s’) 3. Transforms judgment (q0,s) X ß u

Outline Motivation Defining “monitor enforces a policy” Which policies can monitors enforce? Defining “monitor enforces a policy” Define policies Define monitors Define enforcement of policies by monitors Delineating the enforceable policies Infinite renewal properties Comparison of renewal, safety, and liveness Summary

Effective Enforcement Effective enforcers adhere to two principles Soundness: Monitors’ outputs (i.e., observable executions) must satisfy the desired property Transparency: Monitors must not alter the semantics of valid inputs Conservative definition: on a valid input sequence s, a monitor must output s Target can expect that if it behaves well, it’s actions will execute properly.

Enforcing Properties Automaton X starting in q0 enforces p on a system with action set A iff "sÎA∞: $uÎA∞: 1. (q0,s) X ß u 2. p(u) [Soundness] 3. p(s) Þ (s=u) [Transparency]

Outline Motivation Defining “monitor enforces a policy” Which policies can monitors enforce? Defining “monitor enforces a policy” Define policies Define monitors Define enforcement of policies by monitors Delineating the enforceable policies Infinite renewal properties Comparison of renewal, safety, and liveness Summary

Enforcement Powers Related Work Previous work has examined the enforcement bounds of monitors [Schneider ’00; Viswanathan ’00; Hamlen, Morrisett, Schneider ’03; Fong ’04] In the previous models, monitors respond to dangerous actions by halting the target outright Our framework considers more expressive monitors that can accept, suppress, and insert actions

Enforcing Properties with Edit Automata Powerful enforcement technique Suppress (feign execution of) potentially bad actions, and later, if the suppressed actions are found to be safe, re-insert them Limitation: model assumes monitors have same computational power as observing system In practice, some actions cannot be feigned: Actions requiring an outside system to execute Time-dependent actions

Renewal Properties Technique of suppressing dangerous actions allows edit automata to enforce any reasonable infinite renewal property Renewal: “Infinite executions are good iff they are good infinitely often” "sÎAω : p(s) Û {u≤s | p(u)} is an infinite set

Example Renewal Property Check subsystem configuration after rebooting: Whenever reboot is called, it must be followed by a call to checkConf Is a renewal property: A valid infinite execution has infinitely many valid prefixes (in which checkConf always follows reboot) An invalid infinite execution has only finitely many valid prefixes There must be some invalid prefix where the next action after a reboot is not checkConf — all successive prefixes are invalid

Edit Automata Enforcement (Lower Bound) Theorem: " properties p such that 1. p is a renewal property, 2. p(●), and 3. "sÎA* : p(s) is decidable, $ an edit automaton that enforces p. Edit automata can enforce any reasonable renewal property

Edit Automata Enforcement (Lower Bound) Proof idea: Technique of suppressing actions until they are known to be safe causes every valid prefix, and only valid prefixes, of the input to be output Given a renewal property p, construct an edit automaton X that uses this technique In all cases, X correctly enforces p If input s has finite length, X outputs longest valid prefix of s Else if Øp(s) and s is infinite, X outputs the longest valid (finite) prefix of s (since p is a renewal property) Else if p(s) and s is infinite, X outputs every prefix of s and only prefixes of s

Edit Automata Enforcement (Precise Bounds) Edit automata enforce exactly the set of reasonable renewal properties, plus some extra “almost renewal” properties Details are in ESORICS ’05 proceedings “Almost renewal” properties may consider some infinite executions to be valid even though they have only finitely many valid prefixes To enforce an “almost renewal” property on an execution u, an automaton needs to decide while inputting u that u is the only valid extension of its current input At that point, the automaton can enter an infinite loop to insert the remaining actions of u

Outline Motivation Defining “monitor enforces a policy” Which policies can monitors enforce? Defining “monitor enforces a policy” Define policies Define monitors Define enforcement of policies by monitors Delineating the enforceable policies Infinite renewal properties Comparison of renewal, safety, and liveness Summary

Definition Reminder Safety: “Bad executions cannot be made good” Liveness: “Finite executions can be made good” Renewal: “Infinite executions are good iff they are good infinitely often”

Example Renewal Property, Revisited Check subsystem configuration after rebooting: Whenever reboot is called, it must be followed by a call to checkConf Is not a safety property: Øp(reboot) but p(reboot;checkConf) Is not a liveness property: reboot;a cannot be “made good” when a¹checkConf

Safety, Liveness, Renewal All Properties 1 File access control 2 Trivial 3 Eventually audit 4 Check configuration after rebooting 5 Termination 6 Termination + File access control Renewal Safety Liveness 1 2 3 5 4 6

Outline Motivation Defining “monitor enforces a policy” Which policies can monitors enforce? Defining “monitor enforces a policy” Define policies Define monitors Define enforcement of policies by monitors Delineating the enforceable policies Infinite renewal properties Comparison of renewal, safety, and liveness Summary

Summary Monitors modeled by edit automata can enforce any reasonable renewal property Renewal properties include all safety properties, some liveness properties, and some properties that are neither safety nor liveness

End Thanks / Questions