1. Jet Propulsion Laboratory The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration The Conundrums of the Costing World Jairus Hihn October 28, 2008 23rd International Forum on COCOMO and Systems/Software Cost Modeling Phoenix SMAP

2. Background The Jet Propulsion Laboratory is a Federally Funded Research & Development Center operated by the California Institute of Technology for the National Aeronautics and Space Administration. As part of the NASA team, JPL enables the nation to explore space for the benefit of humankind by developing robotic space missions to:  Explore our own and neighboring planetary systems.  Search for life beyond the Earth's confines.  Further our understanding of the origins and evolution of the universe and the laws that govern it.  Enable a virtual presence throughout the solar system using the Deep Space Network and evolving it to the Interplanetary Network of the future. 2

3. Measurement Estimation & Analysis “Creating a Quantitative SW Management Culture” Size Growth & Stability Resources & Cost Schedule & Progress Product Quality Estimation & Planning Measuring Development Performance Helping Projects SW Cost Estimation Handbook Software Cost Estimation Class Estimation Support Estimation Tools –Flight SW Cost Model (Team X) –SCAT: Probabilistic COCOMO –SLIC: Code Counter Project SW Measures Guide Managing using Software Metrics Class Measurement Support Software Repositories –Cost –Defects –Foundation Measures SW Engineering Models to support task planning Organizational and Process Measures JPL SW baselines and trends – Cost & Schedule Growth – Productivity – Reuse – Defect Density – Cycle Time MEsA Measure Impact of SQI Process Metrics Benchmarking Hihn1-3

4. Background (cont.) JPL has around 5,000 employees   800 work with software in some capacity SQI has  15 FTE MEsA  4 FTE Assessed at Maturity Level 3 in September 2007  Scope was Class B and C Mission Software 4

5. Cost Estimation Issues: Point 1  All the people with power know just enough to be dangerous  Cost Estimators and Modelers must become change agents 5

6. Jet Propulsion Laboratory The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration Models Work But Getting Buy-in is Still a Struggle

7. OCM Key Points  Organizational Change does not come quickly or easily  The improvement process needs to be approached with many of the same deliberate methods and practices that are used in actual system development.  There are similarities between change at the individual and organizational level  The onus is on us to proactively reach out to customer/others and promulgate change  It is essential to proactively reach out to customers instead of merely waiting for them to come to you. 7

8. The Onus Is On Us  Think about it, what are the alternatives?  Blaming others  Why won’t they (colleagues) use our tool/methodology?  Why don’t those #$@@ managers force them to adopt our new approach?  Repeatedly OCM studies have documented that this approach never works  The Change Agent must accept responsibility for proactively reaching out to ‘customers’ and promulgating change 8

9. Percentage in Each Adopter Category Excerpted from http://www.valuebasedmanagement.net 9

10. OCM Stages Contact Internalization 1 2 3 6 7 COMMITMENT TO CHANGE TIME Adapted from Out from Dependency: Thriving as an Insurgent in a Sometimes Hostile Environment, SuZ Garcia and Chuck Myers, SEPG Conference, 2001 4 5 Institutionalization Mechanisms to support sustaining the change Adoption Mechanisms to support wider rollout of change Initial Use Mechanisms to support measured success in piloting Understanding Mechanisms to assure understanding Awareness Mechanisms to promote awareness We Are Here Hihn1-10

11. Parametric Cost Estimation Evolution YearSW Cost ActivitiesOrganizational Reaction OCM 1986- 1989 Softcost: developed by Tauseworth and Reifer for DSN but not used much Collecting COCOMO metrics for ISS, SCT: probabilistic COCOMO JPL had very limited interest and did not want to be bothered. Contact 1989- 1991 JPL’s first SPI (SORCE) provided funding to collect COCOMO data with effort borken out and detailed write ups Focus on non-NASA projects, later 2 GDS were willing to participate. Awareness 1990- 1995 DSN Implementation Office paid us verify cost using SCT and identify risks, collect metrics, 10 page reports. Begrudging recognition was a good thing Initial Use 1996- 2000 No or little institutional funding, isolated special requests for some larger GDS projects. Cost Growth Study Two of the Mgrs we worked with used models for main estimates Limited Use Awareness in Program Dir. 2001- 2007 SDR requires multiples estimates All day cost and metrics class SCAT is advertised as one of SQI’s best products Team X: Flight SW Cost Model and Chair PSSE position is required More and more managers and reviewers expect to see model based probabilistic estimates. Projects often requested to explain differences. Adoption to Institutionalization Today Growing number of very vocal supporters and independent users of SCAT, especially in Flight SW community SDSPs being revised and include direct references as a requirement for model based estimates Costing is ahead of rest of SQI and we have mostly engaged early majority. Now facing late majority and laggards. Hints of Internalization in early adopters, but they are changing! Early Majority at Initial Use 11

12. Cost Estimation Issues: Point 2  Budget ‘bogies’ get set very early in lifecycle. Sometimes based on casual conversations.  You will typically get held to this number!!  Current proposal and planning process encourages/ demands under-estimating in early stages of lifecycle 12

13. What Should One Cost? Proposed What Flies Performance Index Development Cost 13 In our environment we need to know what it will cost after all the *X!*! happens

14. Cost Estimation Issues: Point 3  We improve our models but rarely do we improve our techniques  Our models have too many inputs  If samples are small not only is the data not normal it is not log normal either 14

15. Key Research Results Part 1 Following is based on research with Tim Menzies using data mining techniques to analyze COCOMO 2004-2008 The results are … Our models have too many inputs –Measures of RE go up with over specified models –Problem is what is best varies Manual stratification does not lead to the ‘best’ model –E.g. a combination of flight SW and Class B ground produces a ‘better’ model then just selecting all your flight records and doing LC. –Nearest Neighbor searches for analogous records based on your current project model inputs 15

16. The results are … The same approach is never best but some combination of the following always wins –Local Calibration –Column Pruning –Row Pruning based on Nearest Neighbor Which is best is determined case-by-case Recalibrating every time you estimate Key Research Results Part 2 16

17. Some Codicil Thoughts There is no such thing as a point estimate that means anything, must account for uncertainty We built SCAT a probabilistic COCOMO in Excel Often the earliest simplest estimate is the best Too often we talk ourselves into unrealistic low costs or get beaten into it. Give Me Multiple high level estimates any day –Detailed estimates get lost in the trees and in an environment of requirements volatility it is a house of cards. Whatever you do be consistent 17