Expert Judgment short course, NIA 15,16 April 2008 UNCERTAINTY 2 day short course on Expert Judgment AMBIGUITY Roger Cooke Resources for the Future Dept. Math, Delft Univ. of Technology April 15,16 2008 INDECISION

schedule 1 DAY 1 Start up 9:00 Ambiguity, Uncertainty, Indecision 9:30 Break 10:30 Expert Judgment for Quantifying Uncertainty 11:00 Expert Judgment Exercise 11:30 Lunch 12:00 Bumper stickers for Expert Judgment 1:00 Break 2:00 How to do an Expert Judgment Study 2:30 Round Table 4:00 DAY 2 Hands on EXCALIBUR 9:00 Stakeholder elicitation exercise 11:00 UNIBALANCE demo 11:30 Lunch 12:00 Utilities in Stakeholders Population 1:00 BREAK 2:30 Hands on UNIBALANCE 3:00 Round Table and evaluation 4:00 AMBIGUITY INDECISION UNCERTAINTY 1

Materials On CD: AMBIGUITY INDECISION UNCERTAINTY 1

Materials In Booklet EJshortcourse1.ppt EJshortcourse2.ppt EJcoursenotes-Classical-Model-Boilerplate EJcoursenotes-Probability-Intro EJcoursenotes-Theory-Rational-Decision EJcoursenotes-Subj-Prob&RelFreq EJcoursenotes-Proper-Scoring-Rules EJcoursenotes-Review-Mathematics-Literature EJcoursenotes-PI-definitions&theorems EJcoursenotes-references AMBIGUITY INDECISION UNCERTAINTY 1

Websites & Links Radiation Protection Dosimetry 90: (2000) http://rpd.oxfordjournals.org/cgi/content/short/90/3/295 NUREG EU Probabilistic accident consequence uncertainty analysis http://www.osti.gov/bridge/basicsearch.jsp http://www.osti.gov/energycitations/basicsearch.jsp EU Probabilistic accident consequence uncertainty assessment using COSYMA http://cordis.europa.eu/fp5-euratom/src/lib_docs.htm RFF workshop expert judgment http://www.rff.org/rff/Events/Expert-Judgment-Workshop.cfm TU Delft Website http://dutiosc.twi.tudelft.nl/~risk/ AMBIGUITY INDECISION UNCERTAINTY 1

UNCERTAINTY How harmful is 100Gy gamma radiation In 1 hr? AMBIGUITY Is John (5’11”) tall? INDECISION Is LOR better than EOR?

AMBIGUITY What Means? INDECISION Whats best? UNCERTAINTY What Is?

UNCERTAINTY Do measurements, Quantify uncertainty AMBIGUITY Define concepts, Domain of application INDECISION Quantify utilities, preferences

Stakeholder/problem owners’ UNCERTAINTY Experts’ job AMBIGUITY Analyst’ job INDECISION Stakeholder/problem owners’ job

can’t remove uncertainty can’t remove uncertainty? Uncertainty Analysis NUREGCR-6545-Earlyhealth-VOL1.pdf

Using Uncertainty to Manage Vulcano risk response Aspinall et al Geol Soc _.pdf AMBIGUITY INDECISION UNCERTAINTY 1

What is Uncertainty. Probability. Fuzzy sets. Degree of possibility What is Uncertainty? Probability? Fuzzy sets? Degree of possibility? Certainty factors? Dempster-Shafer Belief Functions? Mathematical representation: Axioms + Interpretation Interpretation: aka operational definitions epistemic rules rules of correspondence etc etc squizzel.pdf AMBIGUITY INDECISION UNCERTAINTY 1

Operational Definitions The philosophy of science: semantic analysis: Mach, Hertz, Einstein, Bohr A Modern rendering: IF BOB says “The Loch Ness monster exists with degree of possibility 0.0731” to which sentences in the natural language not containing “degree of possibility” is BOB committed? AMBIGUITY INDECISION UNCERTAINTY 1

Objective and Subjective Probability EJCourseNotes-Probability-Intro Objective and Subjective Probability EJCourseNotes-Probability-Intro.doc Probability formalism is Kolmogorov’s axioms, for all events A,B: 0  P(A)  1 P(A’) = 1 – P(A) If A  B =   P(AB) = P(A)+P(B) These can be interpreted either OBJECTIVELY: Limit Relative Frequency, OR SUBJECTIVELY: Partial Belief AMBIGUITY INDECISION UNCERTAINTY 1

Objective: Limit relative frequency Naïve Let A1, A2…be “independent trials of A”, then P(A) = lim (#occurrences in N trials / N) Need probability to define “independent trials” Von Mises (1919) P(outcome i) = lim relative freq of i in a “kollectif” of outcomes, i.e. random sequence Need definition of “random sequence” Kolmogorov, Martin-Lof, Schnorr, etc. (60’s-70’s) Random sequence is one which passes all recursive statistical tests  is not predictable by any “decidable rule”. AMBIGUITY INDECISION UNCERTAINTY 1

Examples N Y Y N N Heads with bent coin: PROBABILITY {Heads} = 3/10: THIS IS RANDOM SEQUENCE Thruster Failure on previous tests Y N PROBABILITY OF FAILURE  3/10: NOT RANDOM SEQUENCE USA wins of previous World Cup Soccer championships N PROBABILITY OF USA WIN  0: Not a RANDOM SEQUENCE

Subjective: Degree of Partial Belief (Ramsey 1926, Borel, DeFinetti 1937, von Neumann & Morgenstern 1944, Savage, 1954) Measure partial belief See if it satisfies axioms of probability AMBIGUITY INDECISION UNCERTAINTY 1

Operational definition: Subjective probability Consider two events: F: France wins next World Cup Soccer tournament US: USA wins next World Cup Soccer tournament. Two lottery tickets: L(F): worth $10,000 if F, worth $100 otherwise L(US): worth $10,000 if US, worth $100 otherwise. John may choose ONE . John's degree belief (F)  John’s degree belief (US) is operationalized as John chooses L(F) in the above choice situation AMBIGUITY INDECISION UNCERTAINTY 1

If your preferences satisfy ‘principals of rationality’: B: Belgium wins next World Cup Soccer tournament. L(F) > L(US); L(US) > L(B);  L(F) > L(B) ?? L(F) > L(US)  L(F or B) > L(US or B) ?? (plus some technical axioms) Then (Fundamental Theorem of Decision Theory) There is a UNIQUE probability P which represents degree of belief: DegBel(F) > DegBel(US)  P(F) > P(US) AND a Utility function, unique op to 0 and 1, that represents values: L(F) > L(US)  Exp’d Utility (L(F)) > Exp’d Utility (L(US)) PROOF (4 hrs) EJCoursenotes-Theory-Rational-Decision.doc AMBIGUITY INDECISION UNCERTAINTY 1

Can subjective probabilities be relative frequencies??? A1, A2…….An…..: yes-no experiments S’pose partial belief independent of order: P{A1=Y, A2= N, A3=N} = P{A1=N, A2= N, A3=Y} THEN (barring pathological case) P(An+1= Y | A1=Y,A2= N, A3=N…An=N) n  #Y / n PROOF: combinatorics (20 min), EJCoursenotes-SubjectiveProb&RelFreq.doc AMBIGUITY INDECISION UNCERTAINTY 1

To clarify Subjective probabilities can = relative frequencies You can be uncertain about a limit rel. frequency You can learn about a rel. freq. thereby reducing your uncertainty You can quantify your uncertainty conditional on, say, X, and be uncertain about X You CANNOT be uncertain about your uncertainty in any other useful sense. “my uncertainty in success is 0.7, but my uncertainty in my uncertainty is 0.5, and my uncertainty in my uncertainty of my uncertainty is 0.3....” DON’T GO THERE AMBIGUITY INDECISION UNCERTAINTY 1

Other interpretations of Probability axioms Classical interpretation (Laplace) ‘ratio of favorable cases to all equi-possible cases’ Logical Interpretation (Keynes, Carnap) ‘partial logical entailment’ Neither were able to provide successful operational definitions.† AMBIGUITY INDECISION UNCERTAINTY 1

Alternative representations of uncertainty Fuzzy sets: many axiomatizations, no operational definitions Degree of Possibility: no operational definitions (see however Eur. J. of Oper. Res. 128, 459-478.p 477). AMBIGUITY INDECISION UNCERTAINTY 1

CAN fuzziness represent uncertainty? μman(Quincy) = μwoman(Quincy) = ½  μman AND woman(Quincy) = Min {μman(Quincy), μwoman(Quincy)} = ½ AMBIGUITY INDECISION UNCERTAINTY 1

Lets have a break AMBIGUITY INDECISION UNCERTAINTY 1