1. AMBIGUITY 2 day short course on Expert Judgment Roger Cooke Resources for the Future Dept. Math, Delft Univ. of Technology April 15,16 2008 UNCERTAINTY INDECISION

2. schedule DAY 1 Start up 9:00 1.Ambiguity, Uncertainty, Indecision 9:30 Break10:30 2.Expert Judgment for Quantifying Uncertainty 11:00 Expert Judgment Exercise11:30 Lunch12:00 3.Bumper stickers for Expert Judgment 1:00 Break 2:00 4.How to do an Expert Judgment Study 2:30 Round Table 4:00 DAY 2 Hands on EXCALIBUR 9:00 Break10:30 Stakeholder elicitation exercise 11:00 UNIBALANCE demo11:30 Lunch 12:00 5.Utilities in Stakeholders Population 1:00 BREAK 2:30 Hands on UNIBALANCE 3:00 Round Table and evaluation 4:00 AMBIGUITY INDECISION UNCERTAINTY 1

3. Materials On CD: AMBIGUITY INDECISION UNCERTAINTY 1

4. Materials In Booklet EJshortcourse1.ppt EJshortcourse2.ppt EJshortcourse3.ppt EJshortcourse4.ppt EJshortcourse5.ppt EJcoursenotes-Classical-Model-Boilerplate EJcoursenotes-Probability-Intro EJcoursenotes-Theory-Rational-Decision EJcoursenotes-Subj-Prob&RelFreq EJcoursenotes-Proper-Scoring-Rules EJcoursenotes-Review-Mathematics-Literature EJcoursenotes-PI-definitions&theorems EJcoursenotes-references AMBIGUITY INDECISION UNCERTAINTY 1

5. Websites & Links Radiation Protection Dosimetry 90: (2000) http://rpd.oxfordjournals.org/cgi/content/short/90/3/295 NUREG EU Probabilistic accident consequence uncertainty analysis http://www.osti.gov/bridge/basicsearch.jsp http://www.osti.gov/energycitations/basicsearch.jsp EU Probabilistic accident consequence uncertainty assessment using COSYMA http://cordis.europa.eu/fp5-euratom/src/lib_docs.htm RFF workshop expert judgment http://www.rff.org/rff/Events/Expert-Judgment-Workshop.cfm TU Delft Website http://dutiosc.twi.tudelft.nl/~risk/ AMBIGUITY INDECISION UNCERTAINTY 1

6. AMBIGUITY Is John (5’11”) tall? INDECISION Is LOR better than EOR? UNCERTAINTY How harmful is 100Gy gamma radiation In 1 hr?

7. AMBIGUITY What Means? INDECISION Whats best? UNCERTAINTY What Is?

8. AMBIGUITY Define concepts, Domain of application INDECISION Quantify utilities, preferences UNCERTAINTY Do measurements, Quantify uncertainty

9. AMBIGUITY Analyst’ job INDECISION Stakeholder/problem owners’ job UNCERTAINTY Experts’ job

10. can’t remove uncertainty? Uncertainty Analysis NUREGCR-6545-Earlyhealth-VOL1.pdf NUREGCR-6545-Earlyhealth-VOL1.pdf

11. Using Uncertainty to Manage Vulcano risk response Aspinall et al Geol Soc _.pdf Aspinall et al Geol Soc _.pdf AMBIGUITY INDECISION UNCERTAINTY 1

12. What is Uncertainty? Probability? Fuzzy sets? Degree of possibility? Certainty factors? Dempster-Shafer Belief Functions? Mathematical representation: Axioms + Interpretation Interpretation: aka operational definitions epistemic rules rules of correspondence etc squizzel.pdf AMBIGUITY INDECISION UNCERTAINTY 1

13. Operational Definitions The philosophy of science: semantic analysis: Mach, Hertz, Einstein, Bohr A Modern rendering: IF BOB says “The Loch Ness monster exists with degree of possibility 0.0731” to which sentences in the natural language not containing “degree of possibility” is BOB committed? AMBIGUITY INDECISION UNCERTAINTY 1

14. Objective and Subjective Probability EJCourseNotes-Probability-Intro.doc EJCourseNotes-Probability-Intro.doc Probability formalism is Kolmogorov’s axioms, for all events A,B: 1.0  P(A)  1 2.P(A’) = 1 – P(A) 3.If A  B =   P(A  B) = P(A)+P(B) These can be interpreted either –OBJECTIVELY: Limit Relative Frequency, OR –SUBJECTIVELY: Partial Belief AMBIGUITY INDECISION UNCERTAINTY 1

15. Objective: Limit relative frequency Naïve –Let A 1, A 2 …be “independent trials of A”, then P(A) = lim (#occurrences in N trials / N) Need probability to define “independent trials” Von Mises (1919) –P(outcome i) = lim relative freq of i in a “kollectif” of outcomes, i.e. random sequence Need definition of “random sequence” Kolmogorov, Martin-Lof, Schnorr, etc. (60’s-70’s) –Random sequence is one which passes all recursive statistical tests  is not predictable by any “decidable rule”. AMBIGUITY INDECISION UNCERTAINTY 1

16. Examples NNNNYYNNNYNNYNNNYYNN YYNYYNYNYNNNNNNNNNNN Heads with bent coin: PROBABILITY {Heads} = 3/10: THIS IS RANDOM SEQUENCE Thruster Failure on previous tests PROBABILITY OF FAILURE  3/10: NOT RANDOM SEQUENCE NNNNNNNNNNNNNNNNNNNN USA wins of previous World Cup Soccer championships PROBABILITY OF USA WIN  0: Not a RANDOM SEQUENCE

17. Subjective: Degree of Partial Belief (Ramsey 1926, Borel, DeFinetti 1937, von Neumann & Morgenstern 1944, Savage, 1954) Measure partial belief See if it satisfies axioms of probability AMBIGUITY INDECISION UNCERTAINTY 1

18. Operational definition: Subjective probability Consider two events: F: France wins next World Cup Soccer tournament US: USA wins next World Cup Soccer tournament. Two lottery tickets: L(F): worth $10,000 if F, worth $100 otherwise L(US): worth $10,000 if US, worth $100 otherwise. John may choose ONE. John's degree belief (F)  John’s degree belief (US) is operationalized as John chooses L(F) in the above choice situation AMBIGUITY INDECISION UNCERTAINTY 1

19. If your preferences satisfy ‘principals of rationality’: B: Belgium wins next World Cup Soccer tournament. L(F) > L(US); L(US) > L(B);  L(F) > L(B) ?? L(F) > L(US)  L(F or B) > L(US or B) ?? (plus some technical axioms) Then (Fundamental Theorem of Decision Theory) There is a UNIQUE probability P which represents degree of belief: DegBel(F) > DegBel(US)  P(F) > P(US) AND a Utility function, unique op to 0 and 1, that represents values: L(F) > L(US)  Exp’d Utility (L(F)) > Exp’d Utility (L(US)) PROOF (4 hrs) EJCoursenotes-Theory-Rational-Decision.docEJCoursenotes-Theory-Rational-Decision.doc AMBIGUITY INDECISION UNCERTAINTY 1

20. Can subjective probabilities be relative frequencies??? A 1, A 2 …….A n …..: yes-no experiments S’pose partial belief independent of order: P{A 1 =Y, A 2 = N, A 3 =N} = P{A 1 =N, A 2 = N, A 3 =Y} THEN (barring pathological case) P(A n+1 = Y | A 1 =Y,A 2 = N, A 3 =N…A n =N)  n   #Y / n PROOF: combinatorics (20 min), EJCoursenotes-SubjectiveProb&RelFreq.doc AMBIGUITY INDECISION UNCERTAINTY 1

21. To clarify Subjective probabilities can = relative frequencies You can be uncertain about a limit rel. frequency You can learn about a rel. freq. thereby reducing your uncertainty You can quantify your uncertainty conditional on, say, X, and be uncertain about X You CANNOT be uncertain about your uncertainty in any other useful sense. “my uncertainty in success is 0.7, but my uncertainty in my uncertainty is 0.5, and my uncertainty in my uncertainty of my uncertainty is 0.3....” DON’T GO THERE AMBIGUITY INDECISION UNCERTAINTY 1

22. Other interpretations of Probability axioms Classical interpretation (Laplace) ‘ratio of favorable cases to all equi-possible cases’ Logical Interpretation (Keynes, Carnap) ‘partial logical entailment’ Neither were able to provide successful operational definitions. † AMBIGUITY INDECISION UNCERTAINTY 1

23. Alternative representations of uncertainty Fuzzy sets: many axiomatizations, no operational definitions Degree of Possibility: no operational definitions (see however Eur. J. of Oper. Res. 128, 459-478.p 477). AMBIGUITY INDECISION UNCERTAINTY 1

24. CAN fuzziness represent uncertainty? μ man (Quincy) = μ woman (Quincy) = ½  μ man AND woman (Quincy) = Min {μ man (Quincy), μ woman (Quincy)} = ½ AMBIGUITY INDECISION UNCERTAINTY 1

25. Lets have a break AMBIGUITY INDECISION UNCERTAINTY 1