1. NASA Office of Safety & Mission Assurance IV&V Cost Estimation – A Joint NASA & US Navy Collaboration to Model & Automate the Process Software Assurance Symposium 30 July ~ 1 August 2003 Prepared by: Tom Robinson Research & Engineering Dept. Naval Air Depot Jacksonville, FL RobinsonTE@navair.navy.mil Software Assurance Research Program FY-2003

2. 30 July ~ 1 August 2003CSIP2003-4562 Initiative Description IV&V Cost Estimation – A Joint NASA & US Navy Collaboration to Model and Automate the Process (CSIP2002-46)  The U.S. Navy has joined with National Aeronautics and Space Administration (NASA) in a collaborative effort to model and automate the process of cost estimation for Independent Verification & Validation (IV&V) of software.  The Naval Air Systems Command developed an easy-to-use cost tool for system software and hardware integration testing that uses a bottoms-up work breakdown structure (WBS), combined with statistical regression cost analysis, that yields cost estimates with a high degree of confidence.  The thrust of this Initiative is to develop similar tools for use by NASA on IV&V projects, and ultimately to improve the speed, accuracy, and consistency of NASA software IV&V cost estimating.

3. 30 July ~ 1 August 2003CSIP2003-4563  Gather, Compile, and Analyze Data detailing Existing NASA Software IV&V Activities.  Establish / Validate a Benchmark with Processes and Procedures for Software IV&V in use by Industry, Academia, and other Government Agencies.  Refine NASA Software IV&V Program Management Processes, Interfaces and Tools necessary to Improve the Speed and Accuracy of Software IV&V Program Scheduling, Risk Assessment, and Cost Estimating.  Develop, prototype, and validate new NASA Software IV&V Program Cost Estimating Tools.  Improve and Evolve NASA Software IV&V Business Processes and Techniques. Goals / Objectives

4. 30 July ~ 1 August 2003CSIP2003-4564 Possible ICE Tools Relationship within a Business Model Regular Consistent Reports Cost & Schedule Technical Metrics All data above contained and configured/controlled in a Common Data Depository Criticality Analysis and Risk Assessment IV&V Should-Cost Estimator All S/W Items (Items for IV&V in Red) ICE Tool Spread by FY Standard Contract Contractor ABCDEFABCDEF - IV&V Criteria - Item Sizes, etc - Labor Rates - Timeline Info Process Improvement Project Tracking ‘ Earned Value’ BCWS ACWP BCWP

5. 30 July ~ 1 August 2003CSIP2003-4565  Established a Joint IV&V Cost Estimating (ICE) Team comprised of Personnel from NASA, the Naval Air Systems Command, Software Industry, and Software Engineering Academia  Developed & Delivered IV&V Effort Man-hour Tool (EMT - ProLogic)  Developed the Prototype IV&V Cost Estimating Tool (‘ICE Tool’)  Compiled & Distributed NASA Software IV&V Activity / Task Matrix, corresponding Dictionary, and Standard ‘Mission Class’ Criteria Tables  Established Initial IV&V Cost Estimating Work Breakdown Structure  Developed & Delivered the Pre-Production ICE Tool Demo during this 2003 NASA OSMA SAS  Production ICE Tool in-work; Hosting data tables on the NASA IV&V Tools Lab network server – Production Delivery in October 2003 Major Accomplishments

6. 30 July ~ 1 August 2003CSIP2003-4566 IV&V Cost Estimating (ICE) Navigator Screen

7. 30 July ~ 1 August 2003CSIP2003-4567 Viewing & Adjusting ICE Results $ 574 K $4,563 K $6,152 K Ideal Budget Budget Shortfall (from Min.) Probable Budget Available (4,563) Ideal (6,152) Min.for Cert. (5,137) 25 % IV&V ActivityReq Des Code Test Criticality Analysis 101 65 WBS Telemetry Processing (TMP)5.1.1.1.3 Criticality Analysis/Risk Assessment Hazard Analysis 102 103 104 105 Risk Analysis Independent Risk Assessment IDS/W Item Save Changes Save Adjustment Rationale Reason for change 5 – Return to prioritized summary of all S/W Items 3 – Any adjustment to any IV&V task activity for that S/W Item requests a reason be entered 1 – ICE ‘Run’ ends with a prioritized summary list of S/W Items 4 – Return to IV&V Details for the S/W Item after saving reasons 2 – S/W Item selected by ‘clicking’ Item in prioritized summary list

8. 30 July ~ 1 August 2003CSIP2003-4568 IV&V Cost Estimating (ICE) Tool Walk-Through

9. 30 July ~ 1 August 2003CSIP2003-4569  Continue to collect and refine IV&V Task Details (Man-hours, Scalars) and Mission Class criteria; Define & refine contract cost Reporting requirements to capture future IV&V costs  Survey Department of Defense programs for IV&V cost data to Assist in tuning / validating the NASA IV&V Cost Estimating Tools  Design and develop a statistical IV&V cost / forecast engine to Overlay the ICE Tool bottoms-up, engineering-based WBS cost estimate  Perform full-up Beta-testing of network-based ICE Tools; Analyze & Report findings; Develop “Tuning Utilities” and maintenance ‘Assistant’  Initiate Independent testing of IV&V Cost Estimating Tools; Analyze Test results; Update / improve tools as required Plans for Fiscal Year - 2004 … Transition!

10. 30 July ~ 1 August 2003CSIP2003-45610 The ‘ICE’ Team Tom Robinson (“Robbie”) NAVAIR Research & Engineering (JAX) RobinsonTE@navair.navy.mil John Marinaro NASA IV&V Facility John.Marinaro@nasa.gov Christina Moats NASA IV&V Facility Christina.D.Moats@nasa.gov Ken McGill NASA IV&V Facility Kenneth.McGill@nasa.gov Tim Menzies, PhD WVU SE Research Chair Tim@Menzies.com Hugh Smith The M & T Company Hugh.Smith@cdicorp.com Dan McCaugherty ProLogic Inc. DanM@prologic-inc.com Josiah Devasirvatham Science Applications International Corp Josiah.Devasirvatham@saic.com Tom Marshall Titan System Corp (Averstar Group) Tom.Marshall@titansystemscorp.com@titansystemscorp.com Rich D’Alessio The M & T Company Rich.Dalessio@cdicorp.com … You ? Volunteers …