Realism in Assessment of Effort Estimation Uncertainty: It Matters How You Ask By Magne Jorgensen IEEE Transactions on Software Engineering Vol. 30, No. 4, April 2004 Presented by Debra Dirlam Oct

Effort Estimation Uncertainty How sure are you of this estimate? Managers depend of your estimate and your level of uncertainty about the estimate. –For sureness in manager’s decisions –For bidding on contracts –For project contingency buffers

It matters how you ask How should you frame your request for uncertainty information about an estimate? Obvious wrong way: “You don’t believe that it will take you more than 1700 hours, do you?”

Traditional Framing of the question Estimators are asked to provide the minimum and maximum effort values based in given confidence levels Confidence level usually 90% “What is the minimum and maximum effort and be 90% sure?”

Alternative Framing of the question Estimators are asked to assess the probability of the actual effort being higher or lower than a certain value. “How likely is it that the project will take more than 1700 hours?”

To Prove: Alternate framing provides greater realism and more useful information Traditional: Give me an estimate that you are 90% certain. Alternate: Give me an estimate and tell me your certainty.

Research Steps Step 1 Identify the size of the systematic overconfidence & understand the reasons –Found overconfidence high –When estimates claimed to be 90% confident they actually were only 60% on target –Level of overconfidence supported by other studies –Reasons… later

Research Step 2 Looked at formal effort estimation uncertainty models designed to replace expert judgment –Some models could remove overconfidence at expense of widening the min-max interval –Conclude that current models could not replace expert judgment –More promising approach is to support expert judgment

Research Step 3 Evaluated several strategies for judgment support in student experiments One evaluated the framing variant and gave promising results The experiment was replicated with software professionals

The Software Professionals Experiment 29 experienced software developers & project managers Paid to participate Divided randomly into 2 groups After giving estimate of most likely effort, half were asked “Tell me the interval in which you are 90% confident” And the other half were asked “Tell me the probability that the actual effort will be between 50% to 200% of your estimate”

10 real world software projects were estimated Used expert judgment and an “experience database” of 5 similar projects Feedback given after each estimate Asked to reflect on performance The Software Professionals Experiment - Training

The Software Professionals Experiment – The Estimations 30 software enhancement tasks previously conducted in a large telecom company Estimate of 1 st task was based on an “experience database” of 5 previously completed tasks Estimate of 2 nd task was based on the “experience database” and the feedback of the 1 st task

Results A hit rate similar to average confidence indicates good correspondence Traditional framing shows slow approach to correspondence Alternative framing shows a close to perfect correspondence on all sequence of tasks.

4 th Step – Full Scale Industrial Experiment 2 medium sized Norwegian software development companies No formal estimation process in place All estimates based on expert judgment Company projects and employees were similar 18 months, projects >10 hours < 8 months Projects were independent of each other

Industrial Experiment Design During estimation phase, asked to complete questionnaire on effort estimation uncertainty assessment with either Traditional or Alternative framing. Framing type was randomly chosen with 47 traditional framings and 23 alternative framings for a total of 70 projects Possible for an estimator to have chance to do both framings No feedback

Industrial Experiment Results Results were similar to previous experiment Traditional: 90% confidence corresponded to 74% hit rate Correspondence better in the Alternative framing: 87% hit rate to 88% confidence Analysis of any systematic favoritism: none or against Alternative framing

Discussion of Results The 2 framing provide the same statistical problem Looks how software professionals perceive and perform the uncertainty tasks in the 2 framings –Finds 2 important differences

Differences in How Software Professionals Perceive and Perform in the 2 Framings Seems to be a better fit between Alternative framing and the format of historical estimation data. The Traditional framing requires more complex analytical skill Uncertainty estimates are highly intuitive

Differences in How Software Professionals Perceive and Perform in the 2 Framings Software professionals may have goals other than realism in uncertainty estimates Worry about providing meaningless wide intervals Providing narrower intervals and more confidence evaluates to more skill as mistakenly perceived by managers In Alternative framing the interval is not provided by the estimator and cannot be used in skill evaluation

Limitations Cannot be generalize to all contexts Evaluated only high confidence uncertainty assessments for the Traditional framing and wide ranges for Alternative framing. Results may be different for other values. Not enough realism. Used questionnaires, importance of the role of requestor.

Conclusions The best approach to assessing the uncertainty of effort estimates depends on many factors –Skill of estimators –Availability of information about previous projects –Type of information about the project to be estimated –Other factors

Conclusion - continued The variety of factors does not lead to general laws to govern the assessment The lack of general laws does not mean that all choices are equally good The use of alternative framing is better supported by empirical evidence than the use of traditional framing. Use Alternative framing