UNCERTAINTY IN COST ESTIMATES Joseph B. Kadane Carnegie Mellon University

“Prediction is difficult, especially about the future” Niels Bohr Robert Storm Peterson Yogi Berra Samuel Goldwyn Mark Twain Outline: 1. Uncertainty in general 2. An example in another field 3. Uncertainty in cost estimation 4. Conclusion 2

Uncertainty in General Suppose you are uncertain about the weather in Pittsburgh tomorrow and in particular about whether it will rain, and whether the high temperature will exceed 80° F. Then there are 4 uncertain events: rain and temp > 80°F rain and temp ≤ 80°F no rain and temp > 80°F no rain and temp ≤ 80°F One and only one of the will occur, i.e. they are mutually exclusive and disjoint. 3

Imagine that you can buy or sell tickets that pay $1 if occurs, and $0 if does not occur. At what price would you be willing to buy or sell such a ticket? 1. If you will lose for sure. 2. If you will lose for sure. 3. If you will lose for sure. 4. If you will lose for sure. 4

Theorem 1 You avoid sure loss if and only if your prices satisfy: 1) for all events A. 2) where S is the sure event. 3) if A and B are disjoint. Hence you avoid being a sure loser if and only if your prices obey the laws of probability. 5

Notes: 1. Your prices need not be those of someone else. These prices – probabilities – are personal, or subjective. 2. Avoiding sure loss does not make you a winner. (Indeed, the most absurd beliefs can be expressed using such prices. 3. This interpretation of probability does not require an infinite (or long) sequence of independent and identically distributed repetitions. 6

What about conditional events? Suppose there are tickets that pay $1 if A and B both occur, $0 if A occurs but B does not, and the price is returned if A does not occur. Suppose you would pay for such a ticket. 7

Theorem 2: You avoid sure loss if and only if your prices satisfy 8

Notes: 1. This is the same as the usual definition of conditional probability, except here it is a consequence of a desire to avoid sure loss. 2. When is unconstrained. 9

2. Example from another field: physics Physics is touted as the most exact of sciences, one in which uncertainty is best controlled. However, as we’ll see, experimental physicists routinely assume that the measurements they make have no systematic errors, and are routinely wrong, by hindsight, in that assessment. 10

Systematic errors can arise from: 1) Undetected bias in experimental procedures 2) Computational approximations 3) Errors in auxiliary variables 4) Deficiencies in theoretical assumptions None of these are apparent to the experimenter at the time. 11

Figure 1: The history of measurements of the speed of light, c. 12

Figure 2: History of reported uncertainty in other physical constants 13

Figure 2: (cont’d) 14

Repeating an experiment many times can reduce the “random error” associated with an average, but does nothing for systematic error. The culture of physics has not traditionally encouraged a discussion of the likely magnitude of systematic error. 15

3. Uncertainty in Cost Estimates “A reported value whose accuracy is entirely unknown is worthless” Churchill Eisenhart Why is it that cost estimates of almost anything are so rarely too high, and very often too low? One simple correction is to multiply the cost estimate by. If an interval is required, multiply by where Perhaps it is possible to do better than this with a more thoughtful approach. 16

What are some of the sources of uncertainly in estimating the cost of large, long-term projects? a) Scope creep: things change. New capabilities are requested. Is the cost estimate supposed to estimate 1 The hypothetical cost of the project if no changes in scope are required. 2 The actual cost, including possible future (now unforeseen) changes in scope? 17

b) Technology risk: the technology to do the project may not currently exist. How much may it cost to produce the technology? What happens if the technological issues cannot be resolved? 18

c) Economic and political changes: If cost estimates are in current dollars, what effect may future inflation have? What may be the effects of legislative or regulatory changes? What assumptions underlie the project? What are the consequences if those assumptions are wrong? 19

4. Conclusions A. Give up the idea that cost estimates are anything but somebody’s opinion. B. Best practice is to express your opinion in terms of a probability distribution. C. Be clear about exactly what you are costing out, particularly whether changes in scope are included. D. Think about all the ways you might be wrong, and what the costs under those scenarios look like. 20

References: 1. For a fast read, introduction: D.V. Lindley, Understanding Uncertainty, J. Wiley & Sons. 2. For proof of theorems 1 and 2, and a more technical discussion, my book Principles of Uncertainty, Chapman and Hall, 2011, also free on the web at my website 3. For the physics: M. Henrion and B. Fischoff, “Assessing uncertainty in physical constants,” American Journal of Physics, 54(9), September,