Realism in Assessment of Effort Estimation Uncertainty: It Matters How You Ask By Magne Jorgensen IEEE Transactions on Software Engineering Vol. 30, No.

Slides:



Advertisements
Similar presentations
CHAPTER 15: Tests of Significance: The Basics Lecture PowerPoint Slides The Basic Practice of Statistics 6 th Edition Moore / Notz / Fligner.
Advertisements

Statistics.  Statistically significant– When the P-value falls below the alpha level, we say that the tests is “statistically significant” at the alpha.
1 / 27 CS 709B Advanced Software Project Management and Development Software Estimation - I Based on Chapters 1-3 of the book [McConnell 2006] Steve McConnell,
Department of Industrial Management Engineering 1.Introduction ○Usability evaluation primarily summative ○Informal intuitive evaluations by designers even.
Hypothesis testing Week 10 Lecture 2.
Software Effort Estimation based on Use Case Points Chandrika Seenappa 30 th March 2015 Professor: Hossein Saiedian.
Inference.ppt - © Aki Taanila1 Sampling Probability sample Non probability sample Statistical inference Sampling error.
UNDERSTANDING RESEARCH RESULTS: STATISTICAL INFERENCE © 2012 The McGraw-Hill Companies, Inc.
AM Recitation 2/10/11.
Overview Definition Hypothesis
© 2008 McGraw-Hill Higher Education The Statistical Imagination Chapter 9. Hypothesis Testing I: The Six Steps of Statistical Inference.
Hypothesis testing is used to make decisions concerning the value of a parameter.
Section 9.1 Introduction to Statistical Tests 9.1 / 1 Hypothesis testing is used to make decisions concerning the value of a parameter.
Sampling : Error and bias. Sampling definitions  Sampling universe  Sampling frame  Sampling unit  Basic sampling unit or elementary unit  Sampling.
Testing a Claim I’m a great free-throw shooter!. Significance Tests A significance test is a formal procedure for comparing observed data with a claim.
14. Introduction to inference
Sampling: Theory and Methods
Employment, unemployment and economic activity Coventry working age population by disability status Source: Annual Population Survey, Office for National.
Employment, unemployment and economic activity Coventry working age population by gender Source: Annual Population Survey, Office for National Statistics.
Hypothesis Tests with Proportions Chapter 10. Write down the first number that you think of for the following... Pick a two-digit number between 10 and.
Stat 1510 Statistical Inference: Confidence Intervals & Test of Significance.
Maximum Likelihood Estimator of Proportion Let {s 1,s 2,…,s n } be a set of independent outcomes from a Bernoulli experiment with unknown probability.
Project estimation Biased advice on producing accurate project estimates and managing expectations with stakeholders. Morgan Strong.
+ The Practice of Statistics, 4 th edition – For AP* STARNES, YATES, MOORE Unit 5: Hypothesis Testing.
+ The Practice of Statistics, 4 th edition – For AP* STARNES, YATES, MOORE Chapter 9: Testing a Claim Section 9.1 Significance Tests: The Basics.
Audit Sampling: An Overview and Application to Tests of Controls
Hypotheses tests for means
CHAPTER 17: Tests of Significance: The Basics
Gile Sampling1 Sampling. Fundamental principles. Daniel Gile
Lecture 16 Section 8.1 Objectives: Testing Statistical Hypotheses − Stating hypotheses statements − Type I and II errors − Conducting a hypothesis test.
MATH 2400 Ch. 15 Notes.
+ “Statisticians use a confidence interval to describe the amount of uncertainty associated with a sample estimate of a population parameter.”confidence.
Slide Slide 1 Copyright © 2007 Pearson Education, Inc Publishing as Pearson Addison-Wesley. Overview.
Employment, unemployment and economic activity Coventry working age population by ethnicity Source: Annual Population Survey, Office for National Statistics.
METHODS IN BEHAVIORAL RESEARCH NINTH EDITION PAUL C. COZBY Copyright © 2007 The McGraw-Hill Companies, Inc.
CHAPTER 15: Tests of Significance The Basics ESSENTIAL STATISTICS Second Edition David S. Moore, William I. Notz, and Michael A. Fligner Lecture Presentation.
Hypothesis Testing An understanding of the method of hypothesis testing is essential for understanding how both the natural and social sciences advance.
Copyright © 2006 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill/Irwin 8-1 Chapter Eight Audit Sampling: An Overview and Application.
Example 10.2 Measuring Student Reaction to a New Textbook Hypothesis Tests for a Population Mean.
Lecture PowerPoint Slides Basic Practice of Statistics 7 th Edition.
Experimentation in Computer Science (Part 2). Experimentation in Software Engineering --- Outline  Empirical Strategies  Measurement  Experiment Process.
Business Statistics for Managerial Decision Farideh Dehkordi-Vakil.
Exercise - 1 A package-filling process at a Cement company fills bags of cement to an average weight of µ but µ changes from time to time. The standard.
Copyright © 2013, 2009, and 2007, Pearson Education, Inc. Chapter 10 Comparing Two Groups Section 10.1 Categorical Response: Comparing Two Proportions.
+ The Practice of Statistics, 4 th edition – For AP* STARNES, YATES, MOORE Unit 5: Hypothesis Testing.
Understanding Basic Statistics Fourth Edition By Brase and Brase Prepared by: Lynn Smith Gloucester County College Chapter Nine Hypothesis Testing.
Lecture №4 METHODS OF RESEARCH. Method (Greek. methodos) - way of knowledge, the study of natural phenomena and social life. It is also a set of methods.
Chapter 9 Audit Sampling – Part a.
6.2 Large Sample Significance Tests for a Mean “The reason students have trouble understanding hypothesis testing may be that they are trying to think.”
Uncertainty and confidence Although the sample mean,, is a unique number for any particular sample, if you pick a different sample you will probably get.
CHAPTER 15: Tests of Significance The Basics ESSENTIAL STATISTICS Second Edition David S. Moore, William I. Notz, and Michael A. Fligner Lecture Presentation.
Slide 20-1 Copyright © 2004 Pearson Education, Inc.
+ Homework 9.1:1-8, 21 & 22 Reading Guide 9.2 Section 9.1 Significance Tests: The Basics.
Chapter Nine Hypothesis Testing.
Audit Sampling: An Overview and Application to Tests of Controls
Chapter 9: Testing a Claim
Unit 5: Hypothesis Testing
Part III – Gathering Data
Chapters 20, 21 Hypothesis Testing-- Determining if a Result is Different from Expected.
[ March 9, 2017] [ Bill Bowles, Audit Supervisor]
Consider This… I claim that I make 80% of my basketball free throws. To test my claim, you ask me to shoot 20 free throws. I make only 8 out of 20.
10.1: 2-Proportion Situations
Introduction to Inference
Introduction to Inference
Significance Tests: The Basics
Significance Tests: The Basics
Estimating a Population Proportion
Hypothesis Testing A hypothesis is a claim or statement about the value of either a single population parameter or about the values of several population.
UNDERSTANDING RESEARCH RESULTS: STATISTICAL INFERENCE
Introduction to Research
Presentation transcript:

Realism in Assessment of Effort Estimation Uncertainty: It Matters How You Ask By Magne Jorgensen IEEE Transactions on Software Engineering Vol. 30, No. 4, April 2004 Presented by Debra Dirlam Oct

Effort Estimation Uncertainty How sure are you of this estimate? Managers depend of your estimate and your level of uncertainty about the estimate. –For sureness in manager’s decisions –For bidding on contracts –For project contingency buffers

It matters how you ask How should you frame your request for uncertainty information about an estimate? Obvious wrong way: “You don’t believe that it will take you more than 1700 hours, do you?”

Traditional Framing of the question Estimators are asked to provide the minimum and maximum effort values based in given confidence levels Confidence level usually 90% “What is the minimum and maximum effort and be 90% sure?”

Alternative Framing of the question Estimators are asked to assess the probability of the actual effort being higher or lower than a certain value. “How likely is it that the project will take more than 1700 hours?”

To Prove: Alternate framing provides greater realism and more useful information Traditional: Give me an estimate that you are 90% certain. Alternate: Give me an estimate and tell me your certainty.

Research Steps Step 1 Identify the size of the systematic overconfidence & understand the reasons –Found overconfidence high –When estimates claimed to be 90% confident they actually were only 60% on target –Level of overconfidence supported by other studies –Reasons… later

Research Step 2 Looked at formal effort estimation uncertainty models designed to replace expert judgment –Some models could remove overconfidence at expense of widening the min-max interval –Conclude that current models could not replace expert judgment –More promising approach is to support expert judgment

Research Step 3 Evaluated several strategies for judgment support in student experiments One evaluated the framing variant and gave promising results The experiment was replicated with software professionals

The Software Professionals Experiment 29 experienced software developers & project managers Paid to participate Divided randomly into 2 groups After giving estimate of most likely effort, half were asked “Tell me the interval in which you are 90% confident” And the other half were asked “Tell me the probability that the actual effort will be between 50% to 200% of your estimate”

10 real world software projects were estimated Used expert judgment and an “experience database” of 5 similar projects Feedback given after each estimate Asked to reflect on performance The Software Professionals Experiment - Training

The Software Professionals Experiment – The Estimations 30 software enhancement tasks previously conducted in a large telecom company Estimate of 1 st task was based on an “experience database” of 5 previously completed tasks Estimate of 2 nd task was based on the “experience database” and the feedback of the 1 st task

Results A hit rate similar to average confidence indicates good correspondence Traditional framing shows slow approach to correspondence Alternative framing shows a close to perfect correspondence on all sequence of tasks.

4 th Step – Full Scale Industrial Experiment 2 medium sized Norwegian software development companies No formal estimation process in place All estimates based on expert judgment Company projects and employees were similar 18 months, projects >10 hours < 8 months Projects were independent of each other

Industrial Experiment Design During estimation phase, asked to complete questionnaire on effort estimation uncertainty assessment with either Traditional or Alternative framing. Framing type was randomly chosen with 47 traditional framings and 23 alternative framings for a total of 70 projects Possible for an estimator to have chance to do both framings No feedback

Industrial Experiment Results Results were similar to previous experiment Traditional: 90% confidence corresponded to 74% hit rate Correspondence better in the Alternative framing: 87% hit rate to 88% confidence Analysis of any systematic favoritism: none or against Alternative framing

Discussion of Results The 2 framing provide the same statistical problem Looks how software professionals perceive and perform the uncertainty tasks in the 2 framings –Finds 2 important differences

Differences in How Software Professionals Perceive and Perform in the 2 Framings Seems to be a better fit between Alternative framing and the format of historical estimation data. The Traditional framing requires more complex analytical skill Uncertainty estimates are highly intuitive

Differences in How Software Professionals Perceive and Perform in the 2 Framings Software professionals may have goals other than realism in uncertainty estimates Worry about providing meaningless wide intervals Providing narrower intervals and more confidence evaluates to more skill as mistakenly perceived by managers In Alternative framing the interval is not provided by the estimator and cannot be used in skill evaluation

Limitations Cannot be generalize to all contexts Evaluated only high confidence uncertainty assessments for the Traditional framing and wide ranges for Alternative framing. Results may be different for other values. Not enough realism. Used questionnaires, importance of the role of requestor.

Conclusions The best approach to assessing the uncertainty of effort estimates depends on many factors –Skill of estimators –Availability of information about previous projects –Type of information about the project to be estimated –Other factors

Conclusion - continued The variety of factors does not lead to general laws to govern the assessment The lack of general laws does not mean that all choices are equally good The use of alternative framing is better supported by empirical evidence than the use of traditional framing. Use Alternative framing