1. Computer Science Systematic Testing and Verification of Security Policies Tao Xie Department of Computer Science North Carolina State University https://sites.google.com/site/asergrp/projects/policy Joint Work with Vincent Hu, Rick Khun, and ACTS group (NIST) JeeHyun Hwang, Evan Martin (NCSU), Alex Liu (MSU)

2. Computer Science Motivation Digital information is –Easy to access –Easy to search Sensitive information requires access control mechanisms Security policies are popularly in access control –Access control policies for applications –Firewall policies for networks

3. Computer Science Motivation - cont. How to ensure the correct specification of security policies? –What you specify is what you get, but not necessarily what you want Solution: systematic testing and verification of security policies

4. Computer Science Example Access Control Policy Subjects: Student, Faculty Actions: Assign, Receive Resources: Grades Rule 1: IF (faculty AND assign AND grades) Permit Rule 2: IF (student AND receive AND grades) Permit Rule 3: OTHERWISE Deny

5. Computer Science Policy Verification Verify policy against specified property What properties can you come up for this policy? Rule 1: IF (faculty AND assign AND grades) Permit Rule 2: IF (student AND receive AND grades) Permit Rule 3: OTHERWISE Deny

6. Computer Science Policy Verification Property: student can never assign grades Rule 1: IF (faculty AND assign AND grades) Permit Rule 2: IF (student AND receive AND grades) Permit Rule 3: OTHERWISE Deny Violated with a counterexample request: faculty|student assign grades

7. Computer Science Policy Verification “when the specification language is sufficiently declarative, users have great difficulty providing a duplicate statement of behavior.” --- Shriram Krishnamurthi [RiseandRise 08] Rule 1: IF (faculty AND assign AND grades) Permit Rule 2: IF (student AND receive AND grades) Permit Rule 3: OTHERWISE Deny

8. Computer Science Our Approaches Systematic policy verification –Property inference [POLICY 06, SSIRI 09, DBSec 10] –Property-quality assessment [ACASC 08] –Properties derived from access control models [POLICY 10DE] Systematic policy testing –Structural coverage criteria [ICICS 06] –Fault models/mutation testing [WWW 07] –Test generation [SESS 07] –Policy engine performance [SIGMETRICS 08, TC] –Policy engine correctness [TAV-WEB 08] –Firewall policy testing/fixing [SRDS 08/09, LISA 10] XACML policies XACML engines Firewall policies

9. Computer Science XACML A standard access control policy language used to express access control policies –who can do what when A request/response language used to express –queries about whether access should be allowed (requests) and –answers to those queries (responses) http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=xacml

10. Computer Science XACML Policy Structure http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=xacml policy target rule2 target rule1 cond1cond2 A Policy Set holds other policies or policy sets. A policy is expressed as a set of rules. Rules have targets and a set of conditions that determine if the rule applies to a given request. Both rule and policy combining algorithms exist to reconcile conflicts.

11. Computer Science A Simple Scenario A Subject who wishes to perform an Action on a Resource must do so through a PEP. The PEP forms the XACML request and sends it to the PDP. The PDP checks the request against the Policy and returns an XACML response. The PEP either Permits or Denies access to the resource.

12. Computer Science Software Testing Expected Outputs Test Outputs Test Inputs Program Expected Responses ResponsesRequests Policy Software Testing Policy Testing

13. Computer Science Research Problems and Solutions Test generation - Request generation –Policy Coverage Criteria –Random request generation –Request generation based on change-impact analysis –Mutation testing to assess fault-detection capability Test-result inspection - Response inspection –Request selection and minimization based on structural coverage

14. Computer Science Structural Policy Coverage Criteria policy target rule2 target rule1 cond1cond2 policy covered if target matches rule2 covered if target matches Condition must evaluate to True and False to be covered entirely

15. Computer Science Random Request Generation The example policy: –Subjects: Student, Faculty –Actions: Assign, Receive –Resources: Grades Model the set of attribute values as a vector of bits and randomize the bits Student Faculty Assign Receive Grades 1000001101

16. Computer Science Cirg: Change-Impact Request Generation policy version synthesis policy versions change-impact analysis 3. request generation counter examples requests

17. Computer Science IF (faculty AND assign AND grades) ELSE IF (student AND receive AND grades) Permit ELSE Deny Counter-example –faculty, assign, grades : Permit  Deny Cirg Example Permit Deny

18. Computer Science Rationale: synthesize two versions whose differences are coverage targets All-to-Empty One-to-Empty One-Increment All-to-Minus-One All-to-Change-One-Effect Synthesized Versions

19. Computer Science Margrave – Change-Impact Analysis Tool Faculty (f) can assign (a) grades (g) Students (s) can receive (r) grades (g) [Fisler et al. ICSE 05] Multi-Terminal Decision Diagrams

20. Computer Science Margrave Sample Output 1:/Subject, role, Faculty/ 2:/Subject, role, Student/ 3:/Resource, resource-class, ExternalGrades/ 4:/Resource, resource-class, InternalGrades/ 5:/Action, command, Assign/ 6:/Action, command, View/ 7:/Action, command, Receive/ 8:/Subject, role, TA/ 12345678 { 00010101 N->P 00011001 N->P 00100101 N->P 00101001 N->P 01010101 N->P 01011001 N->P 01100101 N->P 01101001 N->P }

21. Computer Science Software Mutation Testing Test Outputs Test Inputs Program Mutator Mutant Program Mutant Outputs Differ? Mutant Killed! Mutation Operators

22. Computer Science Policy Mutation Testing ResponsesRequests Policy Mutator Mutant Policy Mutant Responses Differ? Mutant Killed! Mutation Operators

23. Computer Science Components of Mutation Testing Framework ResponsesRequests Policy Mutator Mutant Policy Mutant Responses Differ? Mutant Killed! Mutation Operators

24. Computer Science Research Questions Does test selection based on structural coverage criteria produce request sets with high fault-detection capability? What are the individual characteristics of each mutation operator? –Are some more difficult to kill than others? –Are some easily killed by request sets selected based on structural coverage criteria?

25. Computer Science Sample Policies continue: 51 policies, 56 rules

26. Computer Science # of Requests Generated and Selected continue: 373 (cirg), 500 (random), 32 (reduction)

27. Computer Science Coverage Results Random Request SetCirg SubjectPolCovRuleCovCondCovPolCovRuleCovCondCov codeA100 N/A100 N/A codeB100 N/A100 N/A codeC100 N/A100 N/A codeD100 N/A100 N/A conference00N/A100 N/A default-210092.3175100 50 demo-11100 75100 50 demo-26100 50100 50 demo-5100 75100 50 mod-fedora10084.6258.3384.62 33.33 simple-policy100 N/A100 N/A continue: 32% RuleCov (random) vs. 98% RuleCov(cirg)

28. Computer Science Mutation Operators, Mutation, and Equivalent Mutant Detection Response s Requests Policy Mutator Mutant Policy Mutant Response s Differ? Mutant Killed! Mutation Operators

29. Computer Science Mutation Operators Each operator mutates a different policy element: policy set, policy, rule, condition, and/or their associated targets and effects. OpDescription CPCChange Policy Combining algorithm CRCChange Rule Combining algorithm CREChange Rule Effect PSTFPolicy Set Target False PSTTPolicy Set Target True PTFPolicy Target False PTTPolicy Target True RCFRule Condition False RCTRule Condition True RMPRemove Policy RMPSRemove Policy Set RMRRemove Rule RTFRule Target False RTTRule Target True

30. Computer Science IF (faculty AND assign AND grades) ELSE IF (student AND receive AND grades) Permit ELSE Deny The CRE mutation operator is performed on each rule and changes the decision effect (Permit  Deny) Change Rule Effect (CRE) Example Permit Deny

31. Computer Science Equivalent Mutant Detection An equivalent mutant is semantically equivalent although syntactically different than the original policy. They provide no value and waste resources. We use change-impact analysis to detect equivalent mutants and remove them. Op# Mutants # Equivalent CPC128064 CRC231632 CRE9680 PSTF2560 PSTT2560 PTF5790 PTT5790 RCF6700 RCT180 RMP5790 RMPS2560 RMR6700 RTF6700 RTT6390

32. Computer Science Request Evaluation and Mutant Detection Response s Requests Policy Mutator Mutant Policy Mutant Response s Differ? Mutant Killed! Mutation Operators

33. Computer Science Sun’s XACML implementation An open source implementation of the XACML standard in JavaAn open source implementation of the XACML standard in Java Developed by Sun as part of an ongoing project on Internet Authorization in the Internet Security Research GroupDeveloped by Sun as part of an ongoing project on Internet Authorization in the Internet Security Research Group http://sunxacml.sourceforge.net/

34. Computer Science Mutant-Killing Ratios by Subject

35. Computer Science Mutant-Killing Ratio by Operator

36. Computer Science Our Approaches Systematic policy verification –Property inference [POLICY 06, SSIRI 09, DBSec 10] –Property-quality assessment [ACASC 08] –Properties derived from access control models [POLICY 10DE] Systematic policy testing –Structural coverage criteria [ICICS 06] –Fault models/mutation testing [WWW 07] –Test generation [SESS 07] –Policy engine performance [SIGMETRICS 08, TC] –Policy engine correctness [TAV-WEB 08] –Firewall policy testing/fixing [SRDS 08/09, LISA 10] XACML policies XACML engines Firewall policies

37. Computer Science Firewall Policy Structure A Policy is expressed as a set of rules. A Rule is represented as → is a set of An example firewall policy Rule r 1 ’s A range in each field refers

38. Computer Science Structural Coverage Definition Rationale: when the policy part with a fault is not evaluated (i.e., “covered”), the fault is often not exposed. Rule coverage of a policy P by packets T = #rules evaluated by at least one packet in T #rules in P Predicate coverage of a policy P by packets T #predicates evaluated to true or false by T at least once 2 ×#predicates in P Clause coverage of a policy P by packets T #clauses evaluated to true or false by T at least once 2 ×#clauses in P

39. Computer Science Test Packet Generation Our objective: generating packets for achieving high structural coverage 1.Random Packet Generation –Randomly selects values for a packet 2.Packet Generation based on Local Constraint Solving –Considering individual rules in a policy 3.Packet Generation based on Global Constraint Solving –Considering multiple rules in a policy

40. Computer Science Experiments (measuring coverage) Test 14 firewall policies Generate packets by our proposed three techniques Measure structural coverage.

41. Computer Science Experiments (measuring coverage)

42. Computer Science Experiments (measuring fault detection capability) We also used reduced packet sets (maintaining the same level of structural coverage with the corresponding original packet set)

43. Computer Science NCSU/NIST ACPT Architecture GUI AC Model Templates Data Acquisition Policy Generator Static Verification Dynamic Verification Test inputs based on structural or combinatorial coverage User, attribute, resource, role, etc. data GUI allows specification of users, groups, attributes, roles, rules, policies, and resources Generate enforceable policies.xml Generate and evaluate test inputs Verify access control policies API/mechanism to consume/acquire external data related to policies Generate test inputs Test inputs with their evaluated decisions Administrator XACML http://www.nist.gov/itl/csd/set/acpt.cfm

44. Computer Science 44 Property specification in ACPT ACPT

45. Computer Science 45 Verify the property against Policy A, the result return false with counterexample. Static Verification

46. Computer Science 46 Verify the property against Policy B, the result return true. Static Verification (cont.)

47. Computer Science 47 Test Input Generation and Evaluation

48. Computer Science 48 XACML Generation

49. Computer Science Conclusion Systematic policy verification –Property inference [POLICY 06, SSIRI 09, DBSec 10] –Property-quality assessment [ACASC 08] –Properties derived from access control models [POLICY 10DE] Systematic policy testing –Structural coverage criteria [ICICS 06] –Fault models/mutation testing [WWW 07] –Test generation [SESS 07] –Policy engine performance [SIGMETRICS 08, TC] –Policy engine correctness [TAV-WEB 08] –Firewall policy testing/fixing [SRDS 08/09, LISA 10] XACML policies XACML engines Firewall policies

50. Computer Science Questions? https://sites.google.com/site/asergrp/projects/policy https://sites.google.com/site/asergrp/projects/policy