Mental Models of Cost Estimation: A Focus on Uncertainty Assessment Mental Models of Cost Estimation: A Focus on Uncertainty Assessment 24 th International Forum on COCOMO and Systems/Software Cost Modeling Dr. Ricardo Valerdi, MIT Dr. Linda Newnes, University of Bath Dr. Yee-May Goh, University of Bath Melanie Kreye, University of Bath Dr. Jairus Hihn, NASA JPL November 3, 2009

Outline Functions of mental models Example mental model from NASA Mental models of risk assessment Methodology for capturing mental models Measuring change in mental models Implications & next steps

© Northrop Grumman Mission Systems

What is the value of π ? Physics: … Math: c/d COCOMO user: ~3.14 (round up to 4.0) Accounting: 3.14 or whatever you want Sales: 3.14 (before the mail-in rebate) CEO: where’s my cake?

Human Memory Organized around schemas –Abstract and general rules based on experience and other people/information When schemas are initiated, they direct behavior In a novel situation, individuals recall certain schemas in order to make sense of the new information

Mental Models Psychological representations of real, hypothetical or imaginary situations (Craik, K. The Nature of Explanation, 1943). Mental models include what a person thinks is true, not necessarily what is actually true Mental models are similar in structure to the thing or concept they represent Mental models allow a person to predict the results of their actions Mental models are simpler than the thing or concept they represent. They include only enough information to allow accurate predictions

Functions of Mental Models Rouse, W. B., People and organizations: explorations of human-centered design, Wiley 2007.

Example: Estimation Process at NASA (Normative Model) Source: Hihn, J., Griesel, A., Bruno, K., Fouser, T., Tausworthe, R., Mental Models of Software Forecasting, NASA JPL.

Example: Estimation Process at NASA (Descriptive Model) Source: Hihn, J., Griesel, A., Bruno, K., Fouser, T., Tausworthe, R., Mental Models of Software Forecasting, NASA JPL.

Mental Models of Risk Assessment Canadian Standards Association Source: Fischoff, B., Realistic Risk Disclosure in Newly Normal Times.

Risk Assessment and Management Source: Cooper, L. P., How Project Teams Conceive of and Manage Pre-Quantitative Risks, PhD Dissertation, University of Southern California, 2008.

Methodology for Capturing Mental Models Semi-structured interviews (normative) –What models do you use to conceive of risk? –How is risk captured as part of the cost estimation activity? Ethnography (descriptive) –Document descriptive risk perception based on spontaneous responses to certain problem situations –Differentiate risk perceptions across roles (cost estimator, PM, software engineer, etc.) and context (mission critical, IT development, telecom, etc.)

Temporal Properties of Mental Models Immediate PastFuture Source: Radioactive Waste Management Committee, Report No. NEA/RWM/FSC(2003)7/REV1, The Mental Models Approach to Risk Research.

Measuring Change in Mental Models Doyle, J. K., Radzicki, M. J., Trees, W. S., Measuring the effect of systems thinking interventions on mental models, pp in George P. Richardson and John D. Sterman, eds. Proceedings of the 1996 International Conference of the System Dynamics Society. Cambridge, Massachusetts. July 22-25, Attain a high degree of experimental control 2. Separate measurement and improvement 3. Collect data from individuals in isolation 4. Collect detailed data from the memory of each individual 5. Measure change rather than perceived change 6. Obtain quantitative measures of characteristics of mental models 7. Employ a naturalistic task and response format 8. Obtain sufficient statistical power

Way Forward Implications –Documenting mental models will help understand the schemas for pre-quantitative risk assessment –Richness of mental models will allow implicit information to influence quantitative risk assessment –Contextual differences in risk assessment can be made explicit Next steps –Elicitation of mental models from a varied sample of engineering decision makers –Integration of uncertainty models with cost models

Thank You