Costing Secure Systems Workshop Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Costing Secure Systems Workshop Edward Colbert, Danni Wu Yue Chen 5th Workshop on Costing Secure Systems USC CSE Annual Research Review 2004 © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Goal Of Workshop  Review proposed Model for costing development of secure systems Extensions to COCOMO II for development of secure software systems Feedback on behavior analysis Validate proposed models Identify research opportunities Kick-off Delphi process Review possible approaches & forms Identify data sources   © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Participants Ed Colbert, USC, Moderator (ecolbert@cse.usc.com) Barry Boehm, USC, Guru (boehm@cse.usc.com) Cheryl Jones, US Army (cljones@pica.army.mil) Clate Stansbury, MCR (cstansbury@mcri.com) Danni Wu, USC (danwu@usc.edu) David Seaver, Price Systems (david.seaver@pricesystems.com) Lee Bergstrom, Lockheed Martin Management & Data System (lee.bergstrom@lmco.com) Paul Stelling, Aerospace Corp. (stelling@aero.org) Sachin Shaw, USC (sachsh@cse. usc.edu) Sherman Paskett, General Dynamics Decision Systems (sherman.paskett@gd-decisionsystems.com) Winsor Brown, USC, Camera Man (awbrown@cse.usc.edu) Yue Chen, USC (yuec@cse.usc.edu) © 2002-4 USC-CSE 18 January 2019

Highlights Project Schedule Early Estimation Model Work-Breakdown Structure (WBS) Common Criteria & Security Objectives COCOMO Security Extension Delphi Collection Approach Issues To Do © 2002-4 USC-CSE 18 January 2019

Cost Model for System Security Increment 1 (Feb – July ’04) Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Cost Model for System Security Increment 1 (Feb – July ’04) Task Element Activities 1. Develop Early Estimation Model Prototype model 2. Sources of Cost Identify, define, scope sources of cost Relate sources of cost to FAA WBS Recommend type of CER for each 3. Secure Product Taxonomy Identify, define, scope product elements 4. COCOMO II Security Extensions Refine model form and data definitions 5. COCOTS Security Extensions Explore security aspects in COCOTS data collection © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Cost Model for System Security Increment 2 (Aug ’04 – July ’05) Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Cost Model for System Security Increment 2 (Aug ’04 – July ’05) Task Element Activities 1. Develop Early Estimation Model Experimental use & refinement 2. Sources of Cost Prioritize sources of cost needing CER’s Refine, prototype, experiment with top-priority CER’s Relate to scope of COCOMO II security extensions 3. Secure Product Taxonomy Experimental use, feedback, and refinement 4. COCOMO II Security Extensions Refine, scope, form, definitions based on results of Tasks 1-3 Experimentally apply to pilot projects, obtain usage feedback 5. COCOTS Security Extensions Develop initial scope, form, definitions based on results of Tasks 1-4 © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Cost Model for System Security Increment 3 (Apr ’05 – Sep ’06) Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Cost Model for System Security Increment 3 (Apr ’05 – Sep ’06) Task Element Activities 1. Develop Early Estimation Model Evolution; integration with other models 2. Sources of Cost Refine sources of cost, CER’s based on usage feedback Integrate with other models Address lower-priority CER’s as appropriate 3. Secure Product Taxonomy Monitor evolution 4. COCOMO II Security Extensions Baseline model definitions Collect project data Develop initially calibrated model; experiment and refine 5. COCOTS Security Extensions Experimentally apply to pilot projects Refine, baseline based on usage feedback © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Highlights Project Schedule Early Estimation Model Work-Breakdown Structure (WBS) Common Criteria & Security Objectives COCOMO Security Extension Delphi Collection Approach Issues To Do © 2002-4 USC-CSE 18 January 2019

Formula for Cost of System & of Security Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Formula for Cost of System & of Security Etotal = EInitial/Mission Analysis + EInvestment Analysis + ESystem Engineering + EDev & Imp + ESys of Sys Integration + Einstall/deployment + EO&M + EDisposal EDev & Imp = Edesign & build HW + Edesign & build SW + Epurchased services + ECOTS-Sys Etotal (Security) = Etotal (with sec) – Etotal (without sec) COTSYS  Commercial of the Shelf Systems © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Formula Elements & COCOMO Family Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Formula Elements & COCOMO Family Formula Elements COCOMO Family Member ESystem Engineering COSYSMO (new) Edesign & build SW COCOMO-II COCOTS ESys of Sys Integration COSoSIMO (new) © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Draft Model of Cost Distribution System Purchase Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Draft Model of Cost Distribution System Purchase Distribution depends on type system acquired # systems affects Installation O&M Disposition After 1st, costs are less Example: $1M System Numbers for show only CORE of this model is for SW © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Draft Model of Cost Distribution System Development Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Draft Model of Cost Distribution System Development Distribution depends on type system acquired # systems affects Installation O&M Disposition After 1st, costs are less Example: $1M System Numbers for show only CORE of this model is for SW © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Draft Model of Cost Distribution System Development with COTS Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Draft Model of Cost Distribution System Development with COTS Distribution depends on type system acquired # systems affects Installation O&M Disposition After 1st, costs are less Example: $1M System Numbers for show only CORE of this model is for SW © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Draft Model of Cost Distribution Services Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Draft Model of Cost Distribution Services Distribution depends on type system acquired # systems Affects Implementation O&M Disposition After 1st, costs are less Example: $1M System Numbers for show only CORE of this model is for SW © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Cost Model for Secure System Approach Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Cost Model for Secure System Approach Identify major sources of cost To Develop Own Including Facilities Equipment People Acquired Systems Services © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 How to Estimate Costs? Costing Approaches Activity Models Unit Costing Analogy Base Parametric For each source of cost, identify appropriate means Cost Estimation Relation (CER) © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Cost Estimation Relations (CER) Example Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Cost Estimation Relations (CER) Example Sample Activity Preparation for Training Classroom Training Periodic Training on new procedures Software Development CER Activity–based Unit costing Analogy-based Parametric Rule 10-20 hours for each Class Hour N trainers total M trainees It cost us $XXX last year,… COCOMO II © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Highlights Project Schedule Early Estimation Model Work-Breakdown Structure (WBS) Common Criteria & Security Objectives COCOMO Security Extension Delphi Collection Approach Issues To Do © 2002-4 USC-CSE 18 January 2019

FAA Acquisition & Standard WBS Analyzed FAA WBS to identify where security will affect activities David Seaver pointed not clear where Analysis of Security Issues related to data management fits Started study Standard WBS © 2002-4 USC-CSE 18 January 2019

Highlights Project Schedule Early Estimation Model Work-Breakdown Structure (WBS) Common Criteria & Security Objectives COCOMO Security Extension Delphi Collection Approach Issues To Do © 2002-4 USC-CSE 18 January 2019

Map Security Objectives with Common Criteria Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Map Security Objectives with Common Criteria Updates © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Highlights Project Schedule Early Estimation Model Work-Breakdown Structure (WBS) Common Criteria & Security Objectives COCOMO Security Extension Delphi Collection Approach Issues To Do © 2002-4 USC-CSE 18 January 2019

COCOMO II Security Driver (SECU) Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 COCOMO II Security Driver (SECU) Viewpoints Design & Development for Security Operational Security Physical Security (Development Constraints) Driver Ratings Nominal High Very High Extremely High Sky High Stratospheric Sky High (or Super High) Stratospheric (or Ultra High) New COCOMO Levels © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Effect Of Security On COCOMO II Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Effect Of Security On COCOMO II Security functional requirements add to project’s KSLOC’s Many systems add Security Manager or Authorization– & Access-Controller “Core Application” adds code to do support authorization & access checks PMtotal = PMtrusted + PMapplication © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Example of COCOMO Security Extension Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Example of COCOMO Security Extension Assume: Application 1000 KSLOC SECU (App) = Nominal Trusted SW 10 KSLOC SECU (trusted) = High SF = 1 All Multipliers (except Security) = 1 (Nominal) SECU Values Nominal = 1 High = 2 VH = 3 EH = 4 SH = 6 Strat = 7 Then PM (total) = 2.94 * (PM (Trusted) + PM (App)) = 2.94 * (10 * SECU(Trusted)) + (1000 * SECU(App)) = 2.94 * ((10 * 2) + (1000 * 1)) = 2.94 * 1020 Effort in Person Month SF: Scale Factors (5) EM: Effort Multipliers(17) © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Number of Security Drivers Issue Current model Add 1 driver that addresses security from 3 viewpoints Development Operational Physical (Development Constraints) Alternative: 2 drivers Security Functions or Objectives Security Assurance Need further research © 2002-4 USC-CSE 18 January 2019

Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 Effect Of Security On COCOMO II (cont.) Relations to Existing Drivers Pre-Workshop If Security driver rating is >= high & following drivers must be > Nominal PREC Precedence (team done similar systems) PMAT Process Maturity TEAM Team Cohesion RELY Required software reliability CPLX Product complexity DOCU Documentation match to life-cycle needs SITE Multi-site development TOOL Use of software tools ACAP Analyst Capability PCAP Programmer Capability Etc. © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

Effect Of Security On COCOMO II (cont Effect Of Security On COCOMO II (cont.) Refined Relations to Existing Drivers Attendees thought best to treat “Clashes” as risk e.g. Precedence (PREC) Security > High  Project = high risk if PREC < High, and ACAP, PCAP & APEX < High Need further investigation for Security levels above Highs © 2002-4 USC-CSE 18 January 2019

Effect Of Security On COCOMO II (cont Effect Of Security On COCOMO II (cont.) Refined Relations to Existing Drivers For Scale Factors: Need to consider how much security drives entire project Cannot easily assign different values to Trusted Non-trusted software © 2002-4 USC-CSE 18 January 2019

Highlights Project Schedule Early Estimation Model Work-Breakdown Structure (WBS) Common Criteria & Security Objectives COCOMO Security Extension Delphi Collection Approach Issues To Do © 2002-4 USC-CSE 18 January 2019

COCOMO Security Extension Delphi Issues Different Stakeholders Needs Customer: “how much will system cost me?” Includes costs outside scope of developer e.g. Independent Testing & Certification Affects developer schedule Developer: “how much effort to build system?” Conclusion: Separate Delphi for Customer’s costs/effort other then developer’s Developer need to Effort to support Independent Testing & Certification Adjust schedule for time taken by Independent Testing & Certification © 2002-4 USC-CSE 18 January 2019

COCOMO Security Extension Delphi Issues Level of Detail to Collect Costing Secure Software, FAA-IT-ISS R&D Workshop 1/18/2019 COCOMO Security Extension Delphi Issues Level of Detail to Collect Current draft based on Common Criteria Functional Requirements 11 Classes 67 Families 138 Components 250 Elements Assurance Requirements 10 Classes 42 Families 93 Components  21 Classes x 7 levels ~4 questions each Is this 2 much to Ask? Couldn’t find better alternative Did Eliminated Question about schedule for some Separate Customer assurance work from Developer Created some examples NTK © 2002-4 USC-CSE 18 January 2019 © 2002-4 USC-CSE

COCOMO Security Extension Delphi Issues Example: Development Assurance Req. (ADV) Activity Nominal High Very High Extremely High Super High Ultra High Authentication No explicit authentication requirement Single authentication mechanism (FIA_UAU.4) Multiple authentication mechanism (FIA_UAU.5) Re-authenticating (FIA_UAU.6) Authentication failure handling (FIA_AFL.1) FIA_AFL.1 Protected authentication feedback (FIA_UAU.7) Simple trusted acknowledgement between TSF (FPT_SSP.1) Protected authentication feedback and unforgeable authentication (FIA_UAU.7, FIA_UAU.3) Mutual trusted acknowledgement between TSF (FPT_SSP.2) Trusted KSLOCS? 2-4 3-5 Application KSLOC Increase (%) 1% 1.1% 1.2% Effort to Produce Kernel (%) Additional Application Effort (%) Rational / Comments At higher levels, need to be more bug free, more effort for unit design, documentation & testing then predicted  need to have higher RELY & DOC © 2002-4 USC-CSE 18 January 2019

COCOMO Security Extension Delphi Issues Example: Development Assurance Req. (ADV) Activity Nominal High Very High Extremely High Super High Ultra High Requirements Specification Standard No explicit security requirements Informal functional & interface specification (ADV_FSP.1) Fully defined external interfaces (ADV_FSP.2) Informal security policy modeling (ADV_SPM.1) Semi-formal functional specifications (ADV_FSP.3) Semi-formal security policy modeling (ADV_SPM.2) Semi-formal functional specification (ADV_FSP.3) Formal security policy modeling (ADV_SPM.3) Formal functional specification (ADV_FSP.4) Trusted KSLOCS? Application KSLOC Increase (%) Effort to Produce Kernel (%) 10%      50 100 200 250 Additional Application Effort (%) Rational / Comments       © 2002-4 USC-CSE 18 January 2019

Highlights Project Schedule Early Estimation Model Work-Breakdown Structure (WBS) Common Criteria & Security Objectives COCOMO Security Extension Delphi Collection Approach Issues To Do © 2002-4 USC-CSE 18 January 2019

Issues 1 or 2 Drivers? Length of Delphi Different Stakeholders Needs Separate security assurance from functional requirements 1 driver for both Length of Delphi Different Stakeholders Needs Customer: “how much will system cost me” Developer: “how much effort for system” What data Do projects have? Can we get projects to collect © 2002-4 USC-CSE 18 January 2019

Highlights Project Schedule Early Estimation Model Work-Breakdown Structure (WBS) Common Criteria & Security Objectives COCOMO Security Extension Delphi Collection Approach Issues To Do © 2002-4 USC-CSE 18 January 2019

To Do Create website Behavior Analysis Refine models Revise Delphi Commercial Security community Refine models Revise Delphi Collect & analyze data Write Ph.D. Thesis (theses?) © 2002-4 USC-CSE 18 January 2019