UNCLASSIFIED Schopenhauer's Proof For Software: Pessimistic Bias In the NOSTROMO Tool (U) Dan Strickland Dynetics Program Software Support Ashley Mathis THAAD Project Office System Software Engineering

UNCLASSIFIED 2 Overview (U) Background Pessimism in Code Size NOSTROMO Tool NOSTROMO Demo

UNCLASSIFIED 3

4 Background (U) In 2003, TPO and Dynetics presented a methodology on using Monte Carlo simulation and uncertainty in COCOMO II to estimate risk in estimates In 2004, TPO and Dynetics presented updated methodologies to include Auto-Generated Code and other updates to NOSTROMO A new methodology for estimating pessimistic size growth has been incorporated into NOSTROMO

UNCLASSIFIED 5 Boehm Horn Curve (U) [Ref (1) - SOFTWARE COST ESTIMATION WITH COCOMO II (P 10) - BOEHM 2000] Size (and effort) should converge over time relative to development phase During Concept of Operation, estimates can be 50% to 200% of the actual final size Some types of software are always on the “bottom” part of the curve – always growing in size

UNCLASSIFIED 6 Code Size Growth (U) Code size estimates (and subsequent cost/schedule estimates) are required long before software requirements are baselined and finalized Code estimates grow to any number of factors: Requirements Volatility Auto-Generated Code Poor Estimating Techniques Many large, complex programs exhibit constant code size growth outside the boundaries of the “horn curve” We define pessimism as extending the boundaries of the expected code size growth to historical limits with only growth potential – “things can only get worse”

UNCLASSIFIED 7 Three Historical Projects – Size Growth (U) Three projects that have completed development or are very near completion Graphs represent code size estimates over time for new code and subsequent Effective SLOC Graphs show growth realization in new and ESLOC growth over time

UNCLASSIFIED 8 Three Historical Projects – Percentage Size Growth (U) Projects new code size grew % from original estimates Projects ESLOC size grew % from original estimates Programs exhibited extensive growth after Critical Design Review (CDR) – during Coding and Unit Testing Percentage growth is outside of the boundaries expected by the horn curve

UNCLASSIFIED 9 Pessimism Curve (U) Plot the data points of the three projects in the same manner as the relative size of the horn curve Assume that the “ceiling” for Pessimism is the final size x and the “floor” encompasses the most extreme data points Where data is unavailable, use the horn curve values and some extrapolation if necessary * ** * Value taken from horn curve – no like value in data ** Value extrapolated – average of previous and subsequent phases

UNCLASSIFIED 10 Smoothed Pessimism Curve (U) Mapping the points of the pessimism curve against the horn curve show an exponential curve of greater growth potential Using MS_Excel, we added an exponential trendline to smooth the pessimism curve and identify a formula for use (R-squared value = ) Using the smoothed curve, we can calculate the ranges for our pessimism curve as tied to development phase We used this new range (inverted) as boundary conditions for size in NOSTROMO NOSTROMO uses a triangle distribution between high and low boundaries for size

UNCLASSIFIED 11 Original NOSTROMO Concept (U) NOSTROMO first takes normal COCOMO II inputs NOSTROMO accounts for uncertainty in the settings NOSTROMO uses Monte Carlo to simulate hundreds of COCOMO II estimates using the inputs and uncertainties NEWT – NOSTROMO Entry Writing Tool – a Delphi polling tool that captures Uncertainty NEWT DATA PAGE REPORTS NOSTROMO Notional Obscurity STatistical Risk Observation MOdel

UNCLASSIFIED 12 New NOSTROMO Concept (U) NEWT DATA PAGE REPORTS NOSTROMO DEFAULT UNCERTAINTY COMBINATION AGC CALCULATOR NOSTROMO offers: multiple methods for inputting Uncertainty combination of subcomponents more reporting capability AGC methodology Pessimistic SLOC estimation PESSIMISM

UNCLASSIFIED 13 Distributions of Uncertainty (U) Setting n-1 Setting n Setting n+1 Level 1 - Certain Level 2 - Low Uncertainty - Normal Level 3 - Medium Uncertainty - Triangle Level 4 - High Uncertainty - Uniform NOSTROMO assumes ceiling and floor limits of the highest and lowest default values for each COCOMO II Scale Factor and Cost Driver NOSTROMO changes the distribution and sets the high and low points based on the level of uncertainty with each factor

UNCLASSIFIED 14 NOSTROMO Application (U) Microsoft Excel 2002 worksheet with attached macros and Visual Basic code Currently on version (as of 10/19/06) Does not use outside applications for Monte Carlo simulation (standard random number generation from Visual basic) Limited testing performed on a proof-of-concept application Suggest limiting Monte Carlo run size to runs Generates two pages of output – Data and Charts Data page contains all outputs from Monte Carlo simulation, charting data, and histograms Charts page contains Confidence Intervals for output of COCOMO II and Putnam models, charts, and histograms NOSTROMO – Data Sheet and CM Control Sheet are Developer pages and should not be removed

UNCLASSIFIED 15 NOSTROMO Input Sheet – Notional Data (U) Software Item Name Scale Factors Code Size Cost Drivers Iterations Putnam Productivity Function Buttons Conditional Formatting Pessimism Toggle Pessimism toggle automatically sets the Requirements Evolution and Volatility to 0% to avoid double counting

UNCLASSIFIED 16 NOSTROMO Pessimism Worksheet (U) Allows the user to develop their own relative size numbers for Pessimism and use in NOSTROMO

UNCLASSIFIED 17 NOSTROMO Outputs – 5000 SLOC (U) 5000 SLOC – Nominal REVL – No Pessimism 5000 SLOC – Pessimism On Pessimism accounts for an 18% increase in Size in the 50 th Percentile NOSTROMO Run

UNCLASSIFIED 18 NOSTROMO Outputs – SLOC (U) 500 KSLOC – Nominal REVL – No Pessimism 500 KSLOC – Pessimism On Pessimism accounts for a 20% increase in Size in the 50 th Percentile NOSTROMO Run

UNCLASSIFIED 19 Demonstration

UNCLASSIFIED 20 Future NOSTROMO Additions and Efforts (U) Addition of Exception Conditions from Ray Madachy’s Expert COCOMO Transition to an application environment (MS-Access or.NET) Expansion of the NOSTROMO tool to address other COCOMO Family models with uncertainties  COSYSMO – especially relevant in development of complex DoD systems  COSOSIMO – System of Systems models are becoming increasingly prevalent in DoD efforts Expansion of the NOSTROMO methodology and tool into Readiness Level models  Software Readiness Levels (SWRL)  Technology Program Management Model (TPMM)

UNCLASSIFIED 21 Conclusion (U) Analysis of size growth potential gives valuable insight into the true risks of software development Software size in many development environments has a tendency to grow, even late into the project development Software estimates need to address any pessimistic growth potentials and software estimators have an obligation to report the risks associated with developments that continue to grow in size New NOSTROMO methodology incorporates pessimistic sizing data to predict the worst-case scenario