1. Expert Judgment short course, NIA 15,16 April 2008
UNCERTAINTY
2 day short course on Expert Judgment
AMBIGUITY
Roger Cooke
Resources for the Future
Dept. Math, Delft Univ. of Technology
April 15,
INDECISION

2. schedule 1 DAY 1 Start up 9:00 Ambiguity, Uncertainty, Indecision 9:30
Break :30
Expert Judgment for Quantifying Uncertainty :00
Expert Judgment Exercise 11:30
Lunch :00
Bumper stickers for Expert Judgment :00
Break :00
How to do an Expert Judgment Study 2:30
Round Table :00
DAY 2
Hands on EXCALIBUR :00
Stakeholder elicitation exercise :00
UNIBALANCE demo :30
Lunch :00
Utilities in Stakeholders Population :00
BREAK :30
Hands on UNIBALANCE :00
Round Table and evaluation :00
AMBIGUITY
INDECISION
UNCERTAINTY 1

3. Materials
On CD:
AMBIGUITY
INDECISION
UNCERTAINTY 1

4. Materials In Booklet EJshortcourse1.ppt EJshortcourse2.ppt
EJcoursenotes-Classical-Model-Boilerplate
EJcoursenotes-Probability-Intro
EJcoursenotes-Theory-Rational-Decision
EJcoursenotes-Subj-Prob&RelFreq
EJcoursenotes-Proper-Scoring-Rules
EJcoursenotes-Review-Mathematics-Literature
EJcoursenotes-PI-definitions&theorems
EJcoursenotes-references
AMBIGUITY
INDECISION
UNCERTAINTY 1

5. Websites & Links Radiation Protection Dosimetry 90: (2000)
NUREG EU Probabilistic accident consequence uncertainty analysis
EU Probabilistic accident consequence uncertainty assessment using COSYMA
RFF workshop expert judgment
TU Delft Website
AMBIGUITY
INDECISION
UNCERTAINTY 1

6. UNCERTAINTY
How harmful is
100Gy gamma radiation
In 1 hr?
AMBIGUITY
Is John (5’11”) tall?
INDECISION
Is LOR better than EOR?

7. AMBIGUITY
What Means?
INDECISION
Whats best?
UNCERTAINTY
What Is?

8. UNCERTAINTY
Do measurements,
Quantify uncertainty
AMBIGUITY
Define concepts,
Domain of application
INDECISION
Quantify utilities,
preferences

9. Stakeholder/problem owners’
UNCERTAINTY
Experts’ job
AMBIGUITY
Analyst’ job
INDECISION
Stakeholder/problem owners’
job

10. can’t remove uncertainty
can’t remove uncertainty? Uncertainty Analysis NUREGCR-6545-Earlyhealth-VOL1.pdf

11. Using Uncertainty to Manage Vulcano risk response Aspinall et al Geol Soc _.pdf
AMBIGUITY
INDECISION
UNCERTAINTY 1

12. What is Uncertainty. Probability. Fuzzy sets. Degree of possibility
What is Uncertainty? Probability? Fuzzy sets? Degree of possibility? Certainty factors? Dempster-Shafer Belief Functions?
Mathematical representation: Axioms + Interpretation
Interpretation: aka
operational definitions
epistemic rules
rules of correspondence
etc etc
squizzel.pdf
AMBIGUITY
INDECISION
UNCERTAINTY 1

13. Operational Definitions
The philosophy of science: semantic analysis: Mach, Hertz, Einstein, Bohr
A Modern rendering:
IF BOB says
“The Loch Ness monster exists with degree of possibility ”
to which sentences in the natural language not containing “degree of possibility” is BOB committed?
AMBIGUITY
INDECISION
UNCERTAINTY 1

14. Objective and Subjective Probability EJCourseNotes-Probability-Intro
Objective and Subjective Probability EJCourseNotes-Probability-Intro.doc
Probability formalism is Kolmogorov’s axioms, for all events A,B:
0  P(A)  1
P(A’) = 1 – P(A)
If A  B =   P(AB) = P(A)+P(B)
These can be interpreted either
OBJECTIVELY: Limit Relative Frequency, OR
SUBJECTIVELY: Partial Belief
AMBIGUITY
INDECISION
UNCERTAINTY 1

15. Objective: Limit relative frequency
Naïve
Let A1, A2…be “independent trials of A”, then P(A) = lim (#occurrences in N trials / N)
Need probability to define “independent trials”
Von Mises (1919)
P(outcome i) = lim relative freq of i in a “kollectif” of outcomes, i.e. random sequence
Need definition of “random sequence”
Kolmogorov, Martin-Lof, Schnorr, etc. (60’s-70’s)
Random sequence is one which passes all recursive statistical tests  is not predictable by any “decidable rule”.
AMBIGUITY
INDECISION
UNCERTAINTY 1

16. Examples N Y Y N N Heads with bent coin:
PROBABILITY {Heads} = 3/10: THIS IS RANDOM SEQUENCE
Thruster Failure on previous tests Y N
PROBABILITY OF FAILURE  3/10: NOT RANDOM SEQUENCE
USA wins of previous World Cup Soccer championships N
PROBABILITY OF USA WIN  0: Not a RANDOM SEQUENCE

17. Subjective: Degree of Partial Belief (Ramsey 1926, Borel, DeFinetti 1937, von Neumann & Morgenstern 1944, Savage, 1954)
Measure partial belief
See if it satisfies axioms of probability
AMBIGUITY
INDECISION
UNCERTAINTY 1

18. Operational definition: Subjective probability
Consider two events:
F: France wins next World Cup Soccer tournament
US: USA wins next World Cup Soccer tournament.
Two lottery tickets:
L(F): worth $10,000 if F, worth $100 otherwise
L(US): worth $10,000 if US, worth $100 otherwise.
John may choose ONE .
John's degree belief (F)  John’s degree belief (US)
is operationalized as
John chooses L(F) in the above choice situation
AMBIGUITY
INDECISION
UNCERTAINTY 1

19. If your preferences satisfy ‘principals of rationality’:
B: Belgium wins next World Cup Soccer tournament.
L(F) > L(US); L(US) > L(B);  L(F) > L(B) ??
L(F) > L(US)  L(F or B) > L(US or B) ??
(plus some technical axioms)
Then (Fundamental Theorem of Decision Theory)
There is a UNIQUE probability P which represents degree of belief:
DegBel(F) > DegBel(US)  P(F) > P(US)
AND a Utility function, unique op to 0 and 1, that represents values:
L(F) > L(US)  Exp’d Utility (L(F)) > Exp’d Utility (L(US))
PROOF (4 hrs) EJCoursenotes-Theory-Rational-Decision.doc
AMBIGUITY
INDECISION
UNCERTAINTY 1

20. Can subjective probabilities be relative frequencies???
A1, A2…….An…..: yes-no experiments
S’pose partial belief independent of order:
P{A1=Y, A2= N, A3=N} = P{A1=N, A2= N, A3=Y}
THEN (barring pathological case)
P(An+1= Y | A1=Y,A2= N, A3=N…An=N) n  #Y / n
PROOF: combinatorics (20 min),
EJCoursenotes-SubjectiveProb&RelFreq.doc
AMBIGUITY
INDECISION
UNCERTAINTY 1

21. To clarify Subjective probabilities can = relative frequencies
You can be uncertain about a limit rel. frequency
You can learn about a rel. freq. thereby reducing your uncertainty
You can quantify your uncertainty conditional on, say, X, and be uncertain about X
You CANNOT be uncertain about your uncertainty in any other useful sense.
“my uncertainty in success is 0.7, but my uncertainty in my uncertainty is 0.5, and my uncertainty in my uncertainty of my uncertainty is ” DON’T GO THERE
AMBIGUITY
INDECISION
UNCERTAINTY 1

22. Other interpretations of Probability axioms
Classical interpretation (Laplace) ‘ratio of favorable cases to all equi-possible cases’
Logical Interpretation (Keynes, Carnap) ‘partial logical entailment’
Neither were able to provide successful operational definitions.†
AMBIGUITY
INDECISION
UNCERTAINTY 1

23. Alternative representations of uncertainty
Fuzzy sets: many axiomatizations, no operational definitions
Degree of Possibility: no operational definitions
(see however Eur. J. of Oper. Res. 128, p 477).
AMBIGUITY
INDECISION
UNCERTAINTY 1

24. CAN fuzziness represent uncertainty?
μman(Quincy) = μwoman(Quincy) = ½ 
μman AND woman(Quincy) =
Min {μman(Quincy), μwoman(Quincy)} = ½
AMBIGUITY
INDECISION
UNCERTAINTY 1

25. Lets have a break
AMBIGUITY
INDECISION
UNCERTAINTY 1