1. 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Harvesting Reference Points for Cost Estimation: A Step in the SEI’s Cost Estimation Method Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 Robert W. Stoddard Dennis Goldenson Rhonda Brown 24 October 2013

2. 2 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Copyright 2013 Carnegie Mellon University This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense. References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHEDON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This material has been approved for public release and unlimited distribution except as restricted below. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. DM-0000618

3. 3 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Introduction QUELCE (Quantifying Uncertainty for Early Lifecycle Cost Estimation) is a multi-year research project led by the Software Engineering Measurement and Analysis (SEMA) team within the SEI Software Solutions Division. Research team membership comprises SEI technical staff with cost estimation background in collaboration with several external faculty (Dr. Ricardo Valerdi, Univ of Arizona, & Dr. Eduardo Miranda, CMU). This research is motivated by (1) the WSARA Act requiring cost estimates pre-Milestone A and (2) DoD’s need for more accurate cost estimation methods that provide continuous monitoring of changing assumptions and constraints.

4. 4 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Mission / CONOPS Capability-based analysis... KPP selection Systems design Sustainment issues... Production quantity Acquisition mgt Scope definition/responsibility Contract award Technology Development Strategy Operational Capability Trade-offs System Characteristics Trade-offs Proposed Material Solution & Analysis of Alternatives Information from Analogous Programs/Systems Program Execution Change Drivers Probabilistic Modeling (BBN) & Monte Carlo Simulation Expert Judgments Information Flow for Early Lifecycle Estimation Plans, Specifications, Assessments Analogy Parametric Models Cost Estimates Program Execution Scenarios with Conditional Probabilities of Drivers/States Driver States & Probabilities Engineering CERs

5. 5 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Repository: Analyze Existing Data to Model Program Execution Uncertainties – 1 Materiel Solution Analysis Phase – Pre-Milestone Estimate A Program Change Repository ProgStateDriver DDG51 cond 1CONOPS cond 2System cond 3CapDef JTRScond 1InterOp cond 2Produc F22cond 1Contract cond 2Function cond 3CONOPS For C2 systems, how often does Strategic Vision change? Records show that Strategic Vision changed in 45% of the MDAPS Driver State Matrix The Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision that will include allied participation. Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs). These changes lead to changes in Capability Definition. Repository identifies probability of change in MDAP cost drivers. 1. Identify Change Drivers & States 3. Assign Conditional Probabilities to BBN Model 4. Calculate Cost Factor Distributions for Program Execution Scenarios 5. Monte Carlo Simulation to Compute Cost Distribution 2. Reduce Cause and Effect Relationships via Dependency Structure Matrix Techniques

6. 6 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Repository: Analyze Existing Data to Model Program Execution Uncertainties – 2 Materiel Solution Analysis Phase – Pre-Milestone Estimate A If Strategic Vision changes, what else changes? 70% of the time the Mission/CONOPS changes Driver State Matrix DSM Cause-Effect Matrix Program Change Repository ProgStateDriver DDG51cond 1CONOPS cond 2System De cond 3CapDef JTRScond 1InterOpera cond 2Production F22cond 1Contract cond 2Functional cond 3CONOPS The Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision that will include allied participation. Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs). These changes lead to changes in Capability Definition. 1. Identify Change Drivers & States 3. Assign Conditional Probabilities to BBN Model 4. Calculate Cost Factor Distributions for Program Execution Scenarios 5. Monte Carlo Simulation to Compute Cost Distribution 2. Reduce Cause and Effect Relationships via Dependency Structure Matrix Techniques

7. 7 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Repository: Analyze Existing Data to Model Program Execution Uncertainties – 3 Materiel Solution Analysis Phase – Pre-Milestone Estimate A Driver State Matrix DSM Cause-Effect Matrix BBN Model When both Strategic Vision & Mission/CONOPs experience change, the BBN calculates that Capability Definition will also change 95% of the time. The Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision that will include allied participation. Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs). These changes lead to changes in Capability Definition. 1. Identify Change Drivers & States 3. Assign Conditional Probabilities to BBN Model 4. Calculate Cost Factor Distributions for Program Execution Scenarios 5. Monte Carlo Simulation to Compute Cost Distribution 2. Reduce Cause and Effect Relationships via Dependency Structure Matrix Techniques

8. 8 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Develop Efficient Techniques to Calibrate Expert Judgment of Program Uncertainties Solution Calibrated Un-Calibrated Estimate of SW Size Program Change Repository 1)Size of ground combat vehicle targeting feature xyz in 2002 consisted of 25 KSLOC Ada 2)Size of Army artillery firing capability feature abc in 2007 consisted of 18 KSLOC C++ 3)… Step 1: Virtual training using reference points Step 2: Iterate through a series of domain- specific tests Step 3: Feedback on test performance Outcome: Expert renders calibrated estimate of size Calibrated = more realistic size and wider range to reflect true expert uncertainty Used with permission from Douglas Hubbard Copyright HDR 2008 dwhubbard@hubbardresearch.com

9. 9 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Research Challenge QUELCE team found the “mining” of domain reference points challenging in terms of requisite knowledge, effort, and schedule. The use of an NVivo-like platform was hypothesized to address the intermediate step of identifying potential change driver experiences documented in historical MDAP artifacts without losing the context of a given artifact, enabling queries by attributes across MDAPs, and achieving review by others prior to committing to a refined domain reference point statement

10. 10 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University How CAQDAS May Help Computer-Assisted/Aided Qualitative Data AnalysiS (CAQDAS) is the “use of computer software to aid qualitative research such as transcription analysis, coding and text interpretation, recursive abstraction, content analysis, discourse analysis, grounded theory methodology, etc.” 1 CAQDAS software initially developed for social science researchers needed a way to assemble, annotate, and make sense out of a myriad of textual artifacts. The QUELCE team conceived of the use of CAQDAS in the sense that “research” has always been inherent in the use of analogy for cost estimation. CAQDAS enables a more formal treatment of that activity. 1 Wikipedia, http://en.wikipedia.org/wiki/CAQDAShttp://en.wikipedia.org/wiki/CAQDAS

11. 11 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University NVivo Experiment Phase 1: Domain experts coded an assigned set of DoD program artifacts likely to exhibit potential change drivers Checkpoint synchronization in the middle of this phase Phase 2: Domain experts and the research team conducted queries against the artifacts using NVivo text analytic and search capabilities Phase 3: Feedback gathered from all regarding NVivo usage On the NVivo experience On high-value documents On the co-occurrence and/or cascading of change drivers

12. 12 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Experimental Procedure Approximately 200 artifacts of an inventory of 1,500 artifacts were identified for coding. We operated in standalone mode, with each domain expert assigned a unique NVivo project on the server. Each NVivo project possessed a unique subset of the 200 artifacts (we balanced the workload among 7 experts). Experts were asked to open assigned artifacts, one at a time, within NVivo and identify the apparent change drivers, if any. We asked that experts to track their time in hours during the coding so that we could gauge the productivity rate in forecasting future coding projects.

13. 13 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Detailed Experimental Work Flow STEP 1: Open NVivo project STEP 2: Open a file to code STEP 4: Sequentially identify change events STEP 5: Within a change event, identify one or more change instances STEP 6: For each change instance, highlight the text and tag with a Change Driver code STEP 7: For each change instance, optionally highlight and tag separate elements of Stimulus, Response, and Outcome STEP 3: Peruse the entire artifact for context

14. 14 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University An NVivo Demonstration Platform

15. 15 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University NVivo Coding Artifacts with Change Drivers

16. 16 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University NVivo Summary of Change Driver Coding

17. 17 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University NVivo Simple Query on a Change Driver

18. 18 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University NVivo Search on “Aerospace” and Schedule Change Driver

19. 19 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University NVivo Word Frequency

20. 20 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University NVivo Word Search Cluster Analysis

21. 21 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University NVivo Analysis of Reproducibility

22. 22 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University NVivo Search on Co-occurrence of Change Drivers Within Artifacts

23. 23 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Early Experimental Feedback Although individual NVivo projects may be employed and then merged together, efficiencies may be possible via the team server setup. The NVivo performance across sensitive networks can be troublesome. Installing NVivo with today’s security infrastructure remains quite challenging. Experts found the coding within NVivo to be painless, e.g., as easy as highlight and drag or highlight and click on a pull-down list of change drivers. Queries provide a means to scan a large portfolio of artifacts for key words, change driver codes, co-occurrence of change drivers, etc…

24. 24 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Future Work Need to enlist additional experts to code the same artifact set and evaluate reproducibility. Need to design and implement expert judgment experiments for improvement in judgment “calibration” using a coded NVivo repository as a reference. Need to refine the query results from an extensive NVivo code repository with the aim of producing refined domain reference points. Need to consider the at-scale implications and logistics using a tool such as NVivo, e.g., as a living DoD asset.

25. 25 2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Contact Information Robert W. Stoddard Principal Researcher Software Solutions Division, SEAP Telephone: +1 412-268-1121 Email: rws@sei.cmu.edu U.S. Mail Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA 15213-2612 USA Web www.sei.cmu.edu www.sei.cmu.edu/contact.cfm Customer Relations Email: info@sei.cmu.edu Telephone: +1 412-268-5800 SEI Phone: +1 412-268-5800 SEI Fax: +1 412-268-6257