1. University of Southern California Center for Systems and Software Engineering ©USC-CSSE1 Value Driven Security Threat Modeling for Off The Shelf Software Systems Yue Chen USC Center for Systems and Software Engineering Emails: {yuec, boehm, lshep}@usc.edu Feb 14, 2007 CSSE Annual Research Review Los Angeles, U.S.A.

2. University of Southern California Center for Systems and Software Engineering ©USC-CSSE2 Trends: Increasing Concerns on COTS Vulnerability Increasing COTS usage Data source: [Boehm et al 2003][Standish] Increasing number of COTS vulnerabilities published Data source: [CERT Statistics] Increasing Trend of COTS Based Applications Introduction

3. University of Southern California Center for Systems and Software Engineering ©USC-CSSE3 Prioritizing COTS Vulnerability is Important Significant Loss: CSI/FBI 2006 Survey reported a total loss of $52,494,290 caused by security incidents from 313 organizations Limited resources for security: 47% spend equal or less than 2% in IT security Data source: CSI/FBI Computer Crime and Security Survey 2006 Challenge: How to spend limited amount of security resources smartly? Introduction

4. University of Southern California Center for Systems and Software Engineering ©USC-CSSE4 Current Practices System context neutral, static vulnerability rankings are recommended by authority organizations such as BugTraq/Symantec, CERT, ISS, NIST, Microsoft, SANS Coarse granularity of rankings: three ~ four levels of rankings (e.g. Critical, Important, Moderate, Low) are used to differentiate 23,620 known COTS vulnerabilities An Real-life Story –USC Ticketing Office System (Source: USC-ISD) Decisions made based on [Butler, 2002] –Best knowledge –Individual experience –Ad hoc Introduction

5. University of Southern California Center for Systems and Software Engineering ©USC-CSSE5 Proposal: Threat Modeling framework based on Attack Path analysis (T-MAP) A novel threat modeling framework for COTS based systems that is sensitive to system stakeholder value context, dynamic, and tool-automated Value Neutral Assessment COTS Vulnerability COTS Vulnerability Rankings Current Approaches : Value Neutral Assessment COTS Vulnerability COTS Vulnerability Rankings Scenario Evaluation Evaluation criteria based on stakeholder value propositions T- MAP: Introduction

6. University of Southern California Center for Systems and Software Engineering ©USC-CSSE6 Permitted Ports Firewall Wrapper Software Applications, COTS e.g. Windows Server 2003 e.g. IIS 6.0 e.g. SQL Server 2000 IT Infrastructure e.g. Web Servere.g. CRM Server Nature of The Problem Org. Values Productivity Reputation e.g. Regulatory Vulnerabilities impacting confidentiality, availability, integrity Attacking Paths Unblocked vulnerabilities Blocked vulnerabilities T-MAP Framework

7. University of Southern California Center for Systems and Software Engineering ©USC-CSSE7 Castle Defense Analog Measure the security of a castle by the value of treasures in the castle, the number of holes on the walls, as well as the size of the holes. T-MAP Framework

8. University of Southern California Center for Systems and Software Engineering ©USC-CSSE8 T-MAP Framework Step 1: Identify key stakeholders and value propositions (the treasures in the castle); Step 2: Establish a set of security evaluation criteria based on stakeholder value propositions; Step 3: Use tool to enumerate and analyze attack paths based on a comprehensive COTS vulnerability database containing 23,620 vulnerability information (the holes); Step 4: Evaluate the severity of each scenario in terms of numeric ratings against the evaluation criteria established in Step 2 (the size of the holes); Step 5: The security threat of each vulnerability is quantified with the total severity ratings of all attack paths that are relevant to this vulnerability; Step 6: System total threat is quantified with the total severity ratings of all attack paths; [Note] Step 3 to 6 are tool automated by the Tiramisu Tool T-MAP Framework

9. University of Southern California Center for Systems and Software Engineering ©USC-CSSE9 Step 1-2: Evaluate Security against Criteria based on Stakeholder/Values Evaluate the severity of security hazard scenarios against stakeholder/value impacts Involves both qualitative and quantitative criteria Technical approach: Figure of Merits and Analytical Hierarchy Process (AHP) –A convenient tool to determine the priority of alternatives in terms of weight through pair-wise comparison –Invented by Dr. Saaty at Business School of Univ. of Pennsylvania in 1970s [Saaty, 1980] –Recommended by Dr. Bodin, Dr. Gordon et al for security investment evaluation [Bodin et al, 2005] T-MAP Framework

10. University of Southern California Center for Systems and Software Engineering ©USC-CSSE10 Using AHP Determine the Weights – An Example  Example (from USC ISD Server X Case Study) T-MAP Framework Value Centric Criteria Possible Breach Scenarios Weights derived through AHP pair-wise comparisons

11. University of Southern California Center for Systems and Software Engineering ©USC-CSSE11 Step 3-4: Enumerate Attack Scenarios Enumerate the scenarios how an attacker can compromise stakeholder values through COTS system vulnerabilities Establish Structured Attack Graph based on a comprehensive COTS vulnerability database involves 23,620 known vulnerabilities reside in 31,713 COTS software T-MAP Framework

12. University of Southern California Center for Systems and Software Engineering ©USC-CSSE12 Step 5: Vulnerability Technical Severity Evaluation Vulnerability Technical Severity Attributes –Impact on confidentiality, integrity and/or availability –Remotely exploitable –Require valid user account on victim host –Needs user activities Based on an emerging national standard [CVSS] by NIST T-MAP Framework

13. University of Southern California Center for Systems and Software Engineering ©USC-CSSE13 Step 6 T-MAP Severity Rating System Severity Weight of Attack Path P: Overall Security Threat Score of COTS System G: ThreatKey of elements in Attack Graph: Effectiveness of Security Practice: T-MAP Framework

14. University of Southern California Center for Systems and Software Engineering ©USC-CSSE14 Security Investment Effectiveness Estimation * Case study results estimated by professional security manager at USC-ITS How much security threats can be avoided by implementing Firewall, Software hardening (patching), user account control, or file system encryption? Results as well depends on the total value of the protected system T-MAP Framework

15. University of Southern California Center for Systems and Software Engineering ©USC-CSSE15 Tool Implementation – Project Code: Tiramisu T-MAP Implementation High-level software architecture How it works?

16. University of Southern California Center for Systems and Software Engineering ©USC-CSSE16 Security Manager’s Ranking vs. Tiramisu Ranking T-MAP Framework

17. University of Southern California Center for Systems and Software Engineering ©USC-CSSE17 Security Economics in Patching Prioritize COTS Based System vulnerabilities under business context –“20% percent of vulnerabilities causes 80% of the security risks”, T-MAP tells what are the 20% Rational: Prioritize vulnerabilities with its ThreatKey; Example screenshot:

18. University of Southern California Center for Systems and Software Engineering ©USC-CSSE18 COTS Security Economics – Finding Sweet Spots Economic curve of security patching (from USC Server X case study) Sweet spot to invest in security Also driven by the total value of system (from USC Server X case study) Sweet spots to invest

19. University of Southern California Center for Systems and Software Engineering ©USC-CSSE19 Limitations Only sensitive to known COTS Vulnerabilities –Empirical study by Arora shows that the average attacks per host per day jumped from 0.31 to 5.45 after vulnerability get published Not sensitive to nuance in local system configurations –Disabled services –Services running on different privileges, etc. Only cover “one-step-attacks” that exploiting COTS vulnerabilities Depends on comprehensive vulnerability database –Our database: 23,260 vulnerability published from 1999-2006 that resides in 31,313 COTS software Cannot effectively address attacks such as Phishing T-MAP Framework

20. University of Southern California Center for Systems and Software Engineering ©USC-CSSE20 Hypothesis Hypothesis #1 *: For given COTS based system whose confidentiality, integrity and availability have different priorities to the stakeholder values, the accuracy of T-MAP results, measured by the Inaccuracy, the ratio of the number of clashes between vulnerability priorities and stakeholder value priorities and the total number of comparisons, will not make any difference comparing to the existing leading approaches (say, the average level of at least three leading approaches, if rating data available). Clash definition: for two technically identical vulnerabilities V A and V B, given the stakeholder value impact severity of V A is higher than V B, if a prioritizing system assign V A less or equal priorities as V B, it is counted as a clash for this prioritizing system. * Hypothesis #1 is a null hypothesis. We aim at disprove them through case studies and tool demos. --X - - -- - -- --- --- Vuln. VBVB VAVA Clash Counting Illustration Research Hypothesis

21. University of Southern California Center for Systems and Software Engineering ©USC-CSSE21 Inaccuracy Comparison Inaccuracy *: the ratio of the number of clashes between vulnerability priorities and stakeholder value priorities and the total number of comparisons * Inaccuracy is a numerical value between 0~1, the larger value indicates the less accuracy For the USC ISD case study, the security manager prioritized 8 vulnerabilities, which translates to 28 comparisons, where 28 = 1+2+… + 7 # of ClashesInaccuracy T-MAP20.071 CVSS60.214 ISS90.321 Hypothesis Validation

22. University of Southern California Center for Systems and Software Engineering ©USC-CSSE22 Contributions A framework to model CBS security –A COTS security evaluation framework that captures stakeholder value propositions –Distill the potential impacts of thousands of vulnerabilities into management friendly numbers at a high-level –Tool automated Has the potential to pro-actively attack security in early life-cycle, instead of taking whatever as is reactively after system built Contribution & Next Steps

23. University of Southern California Center for Systems and Software Engineering ©USC-CSSE23 Discussions and Future Work

24. University of Southern California Center for Systems and Software Engineering ©USC-CSSE24 Thanks!