Software Security Monitors: Theory & Practice David Walker Princeton University (joint work with Lujo Bauer and Jay Ligatti)

July 2003Software Security MonitorsDavid Walker Extensible Systems Plug-in Web Browser Mail Script Mail Server Servlet Web Server Active Router protocol

July 2003Software Security MonitorsDavid Walker Securing Extensible Systems Compile-time/link-time security –policies: memory safety, type safety –tools: type systems, proof-carrying code, model checking Run-time security –policies: access control, resource bounds –tools: access control lists, capabilities, stack inspection

July 2003Software Security MonitorsDavid Walker Run-time Security In general, run-time security properties are enforced by program monitors Abstractly, a program monitor is a process that runs in parallel with an untrusted application –monitors examine application actions decide to allow/disallow application actions may terminate an application, log application actions, reinterpret application actions, etc. –monitors detect, prevent, and recover from erroneous or malicious behavior at run time –generalizes specific enforcement mechanisms such as access control lists, stack inspection, etc.

July 2003Software Security MonitorsDavid Walker Securing Extensible Systems Many questions: –Our application requires property X. Can we enforce it precisely or will we have to get by with an approximation? –How do we write down our policy succinctly and unambiguously? –What specific mechanism will we need to enforce our property? –How do we implement the mechanism?

July 2003Software Security MonitorsDavid Walker Talk Outline Theory: –What is a software security monitor? –What is a security policy and what does it mean to enforce one? –What policies can software security monitors enforce? Language design: –Programming simple policies –Programming complex policies Summary, related work and conclusions

July 2003Software Security MonitorsDavid Walker What is a program monitor? Monitors analyze & transform untrusted application actions: Application generates actions to be input into monitor Monitor Machine executes actions output by monitor a3a1a2 Input Stream a4a2 Output Stream a1 ……

July 2003Software Security MonitorsDavid Walker Possible Monitor Actions Accept the action Halt the application Suppress (skip) the operation Insert some computation Some combination of these

July 2003Software Security MonitorsDavid Walker Formalizing security monitors Program monitors => formal automata that transform a stream of program actions Given: a set of possible program actions A Monitors are deterministic state machines: (Q, q 0, T) where Q= state set q 0 = start state T = transition function

July 2003Software Security MonitorsDavid Walker Operational Semantics  Single step (determined by T):  (S in, q)  (S in ’, q’)  Multi-step (reflexive, transitive closure of T):  (S in, q)  (S in’, q)  Output sequence is observable Input sequences are not observable SoSo SoSo

July 2003Software Security MonitorsDavid Walker A Hierarchy of Security Monitors Insert Suppress OK Halt Truncation Suppression Insertion Edit We classify monitors based on their transformational abilities (ie: based on T).

July 2003Software Security MonitorsDavid Walker An Example: E-Banana.com Set of application actions: A = { take(n), // take n bananas pay(n), // pay for n bananas browse,// browse for bananas receipt// commit } Edit Automaton: take(n)pay(n) pay(n);take(n);receipt pay(n)take(n) receipt tntn pnpn tp n init browse start

July 2003Software Security MonitorsDavid Walker Edit Automata  Definition: (Q,q 0,T) –where T = (t,e,i) –State transition function t t : action x state  state –Emission function e e : action x state  {+,-} –Insertion function i i : action x state  action sequence x state

July 2003Software Security MonitorsDavid Walker Edit Automata Operational Semantics –(S, q)  (S’, q’) if S=a;S’ and t(a,q)=q’ and e(a,q)= + –(S, q)  (S’, q’) if S=a;S’ and t(a,q)=q’ and e(a,q)= - –(S, q)  (S, q’) if S=a;S’ and i(a,q)=(S ins, q’) –(S, q)  (empty, q) otherwise a S ins (E-Accept) (E-Suppress) (E-Insert) (E-Halt)

July 2003Software Security MonitorsDavid Walker Talk Outline Theory: –What is a software security monitor? –What is a security policy and what does it mean to enforce one? –What policies can software security monitors enforce? Language design: –Programming simple policies –Programming complex policies Summary, related work and conclusions

July 2003Software Security MonitorsDavid Walker Security Policies A Security Policy is a predicate P over sequences of actions. Example Policies: –In any program execution, bananas taken equal bananas paid for –Access control, resource bounds policies are properties Non-policies (for our purposes): –Cryptographic uniformity property: The set of all possible outputs of the cryptographic key generation algorithm forms a uniform distribution over the integers –Information-flow policies

July 2003Software Security MonitorsDavid Walker What does it mean to enforce a policy? Principle of Soundness All observable outputs obey the policy  sequences S in.  state q’.  sequence S o 1. (S in, q 0 )  (empty, q’) 2. P(S o ) Principle of Transparency Semantics of executions that already obey policy must be preserved 3. P(S in )  (S in  S o ) SoSo

July 2003Software Security MonitorsDavid Walker Some Useful Equivalences  Remove/Insert unnecessary actions –fclose(f);fclose(f)  fclose(f) Replace a sequence with equivalent actions –socket(S);send(S,m)  socketSend(S,m) Permute independent actions –fopen(f);fopen(g)  fopen(g);fopen(f) Necessary properties: –reflexive, symmetic & transitive –S  S’  P(S)  P(S’)

July 2003Software Security MonitorsDavid Walker E-Banana.com Equivalence Rules: 1) (browse; S)  S 2) (S1; take(n); pay(n); S2)  (S1; pay(n); take(n); S2)

July 2003Software Security MonitorsDavid Walker Conservative Enforcement  Enforcer satisfies Soundness but not necessarily Transparency   properties P. (  sequence S. P(S))  P can be conservatively enforced Conservative

July 2003Software Security MonitorsDavid Walker Effective Enforcement  Enforcer satisfies Soundness and Transparency  Valid sequences can be altered Conservative Effective

July 2003Software Security MonitorsDavid Walker Precise Enforcement  Motivation  In practice, some operations cannot be delayed  Definition  Enforcer satisfies Soundness and Transparency  Enforcer must output actions in lock-step with application Conservative Precise Effective

July 2003Software Security MonitorsDavid Walker Talk Outline Theory: –What is a software security monitor? –What is a security policy and what does it mean to enforce one? –What policies can software security monitors enforce? Language design: –Programming simple policies –Programming complex policies Summary, related work and conclusions

July 2003Software Security MonitorsDavid Walker What properties can be enforced? The enforceable properties depend upon –the definition of enforcement (conservative, effective, precise) –the class of automaton (truncation, suppression, insertion, edit) –the space of possible input programs static program analysis (type systems; proof-carrying code) constrain program execution if the monitor can assume certain “bad” executions do not occur, it can enforce more properties

July 2003Software Security MonitorsDavid Walker Effective Enforcement An E-Banana.com policy: –Our edit automaton is an effective enforcer: It satisfies Soundness It satisfies Transparency Proofs are by induction over the possible inputs –Less powerful automata (truncation, suppression and insertion) cannot enforce the E-Banana property Proof by contradiction shows either Soundness or Transparency will be violated browse*; ((take(n);pay(n) | pay(n);take(n)) ; receipt)*

July 2003Software Security MonitorsDavid Walker A Simple Theorem Theorem: Any decideable predicate P on executions is a property that can be effectively enforced by some edit automaton –Proof: construct a transactional edit automaton that suppresses and logs program actions when ¬P(S) and commits (outputs) when P(S)

July 2003Software Security MonitorsDavid Walker Effectively Enforceable Properties Editing Properties Insertion Properties Suppression Properties Trunc. Prop.

July 2003Software Security MonitorsDavid Walker Talk Outline Theory: –What is a software security monitor? –What is a security policy and what does it mean to enforce one? –What policies can software security monitors enforce? Language design: –Programming simple policies –Programming complex policies Summary, related work and conclusions

July 2003Software Security MonitorsDavid Walker Polymer, the Language Polymer –A domain-specific language for programming security monitors (edit automata) –Java + a couple of simple extensions: simple policy definitions containing –a set of security-relevant actions –security state –decision procedure that produces security “suggestions” (halt, suppress action, insert action, etc) complex policy definitions involving –higher-order policy combinators

July 2003Software Security MonitorsDavid Walker Securing Untrusted Applications Java application policy interface instrumented application describes security- relevant program points contains hooks to call monitor untrusted code separately compiled from policy

July 2003Software Security MonitorsDavid Walker Securing Untrusted Applications Java application policy interface policy implementation instrumented application secure application implements dynamic security policy combines application and policy

July 2003Software Security MonitorsDavid Walker Talk Outline Theory: –What is a software security monitor? –What is a security policy? –What does it mean to enforce a policy? –What policies can software security monitors enforce? Language design: –Programming simple policies –Programming complex policies Summary, related work and conclusions

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { private int openFiles = 0; private int maxOpen = 0; limitFiles(int max) { maxOpen = max; }.... } A Simple Polymer Policy private policy state policy constructor new policy definition extends policy class

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { private int openFiles =... private int maxOpen =... public ActionPattern[] actions = new ActionPattern[] {, };.... } A Simple Polymer Policy Continued set of policy- relevant methods

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { private int openFiles =... private int maxOpen =... public ActionPattern[] actions =... Suggestion before(Action a) { aswitch (a) { case fileOpen(String s) : if (++openFiles <= maxOpen) return Suggestion.OK(); else return Suggestion.Halt(); case fileClose(File f) :... A Simple Polymer Policy Continued policy behavior

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { private int openFiles =... private int maxOpen =... public ActionPattern[] actions =... Suggestion before(Action a) { aswitch (a) { case fileOpen(String s) : if (++openFiles <= maxOpen) return Suggestion.OK(); else return Suggestion.Halt(); case fileClose(File f) :... A Simple Polymer Policy Continued

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy { public ActionPattern[] actions =... private int openFiles =... private int maxOpen =... Suggestion before(Action a) { aswitch (a) { case fileOpen(String s) : if (++openFiles <= maxOpen) return Suggestion.OK(); else return Suggestion.Halt(); case fileClose(File f) :... A Simple Polymer Policy Continued

July 2003Software Security MonitorsDavid Walker Talk Outline Theory: –What is a software security monitor? –What is a security policy? –What does it mean to enforce a policy? –What policies can software security monitors enforce? Language design: –Programming simple policies –Programming complex policies Summary, related work and conclusions

July 2003Software Security MonitorsDavid Walker Complex Monitors Combine simple policies defined over a variety of different resources –eg: sample applet policy file system access control bounds on bytes written and number of files opened restricted network access –no network access after local file is read –communication with applet source only

July 2003Software Security MonitorsDavid Walker Policy Combinators Programmers may write parameterized policy combinators: –And, Or, Forall, Exists, Chinese wall,... s1 s2 AndPolicy: P1P2  s

July 2003Software Security MonitorsDavid Walker Policy Combinators class AndPolicy extends Policy { private Policy p1; private Policy p2; AndPolicy(Policy pol1, Policy pol2) { p1 = pol1; p2 = pol2;... } } first-class policies

July 2003Software Security MonitorsDavid Walker Policy Combinators class AndPolicy extends Policy {... Suggestion before(Action a) { Suggestion s1 = p1.before(a); Suggestion s2 = p2.before(a); if (s1.isOK() && s2.isOK()) return Suggestion.OK(); else... } using suggestions system interprets suggestions at the top level

July 2003Software Security MonitorsDavid Walker Talk Outline Theory: –What is a software security monitor? –What is a security policy? –What does it mean to enforce a policy? –What policies can software security monitors enforce? Language design: –Programming simple policies –Programming complex policies Summary, related work and conclusions

July 2003Software Security MonitorsDavid Walker Future Work Theory –infinite sequences => coinductive proof techniques –resource-bounded programs & monitors time, space and randomness Practice –complete Polymer 1.0 (end of summer) –Polymer evaluation –next up: transactional policies

July 2003Software Security MonitorsDavid Walker Related Work Enforceable security policies –Schneider 00; HMS 02 Monitoring languages –Naccio [ET 99]; Poet and Pslang [ES 99, ES 00]; others –New polymer features: first-class policies & policy combinators, suggestions, abstract actions, formal semantics Aspect-oriented Programming –AspectJ; HyperJ –New polymer features as above –With Dan Dantas, we are developing AspectML

July 2003Software Security MonitorsDavid Walker Summary A general framework for formal reasoning about security monitors –defined a hierarchy of security monitors –gave meaning to the word “enforceable” –developed rigorous proofs concerning enforceable properties Polymer: A programming language for composing security monitors –techniques for modular monitor design & composition –formal semantics as an extension of FeatherWeight Java

July 2003Software Security MonitorsDavid Walker Conclusions Technology for securing extensible systems is in high demand –Software security monitors are one part of the solution For more information, see –Edit Automata: Enforcement Mechanisms for Run-time Security Policies. IJIS –Types and effects for non-interfering program monitors. ISSS 2002 & LNCS –More Enforceable Security Policies. FCS –

July 2003Software Security MonitorsDavid Walker End

July 2003Software Security MonitorsDavid Walker Realistic Monitors Protect complex system interfaces –interfaces replicate functionality in many different places –method parameters communicate information in different forms –eg: Java file system interface 9 different methods to open files 4 different methods to close files filename strings, file objects, self used to identify files

July 2003Software Security MonitorsDavid Walker Abstract Action Definitions java.lang.io FileReader(String fileName); FileReader(File file); RandomAccessFile(...);... FileReader.close(); RandomAccessFile.close();... fileOpen(String n); fileClose();

July 2003Software Security MonitorsDavid Walker Abstract Action Definitions class fileOpen extends ActionSig { boolean canMatch(Action a) { aswitch (a) { case FileReader(_) : return true; case RandomAccessFile () : return true;... } String parameter1(Action a) {.... }

July 2003Software Security MonitorsDavid Walker class limitFiles extends Policy {... Suggestion step(Action a) { aswitch (a) { case fileOpen(String s) :... case fileClose() :... } Abstract Action Pattern Matching fileOpen.canMatch(a) fileOpen.parameter1(a)

July 2003Software Security MonitorsDavid Walker Taxonomy of Precisely Enforceable Properties

July 2003Software Security MonitorsDavid Walker Secure Application Java core Polymer language extensions Host System (Java) Program Monitor Definition Untrusted application

July 2003Software Security MonitorsDavid Walker Policy Architecture: Simple Policies Java core Polymer language extensions Host System (Java) Simple Policy Def. system interface

July 2003Software Security MonitorsDavid Walker Policy Architecture: Abstract Actions Java core Polymer language extensions Host System (Java) Abstract Action Def. concrete system interface abstract system interface Simple Policy Def.

July 2003Software Security MonitorsDavid Walker Policy Architecture: Complex Policies Java core Polymer language extensions Host System (Java) Abstract Action Def. Simple Policy Def. Policy Comb. Def. Complex, System-specific Policy concrete system interface abstract system interface