Click to edit Master title style Click to edit Master subtitle style A PRACTICAL LOOK AT UNCERTAINTY MODELING Steve Unwin Risk & Decision Sciences Group March 7, 2006

2 "The fundamental cause of trouble in the world today is that the stupid are cock-sure while the intelligent are full of doubt.“ Bertrand Russell

3 Illuminating the Path Visual Analytics Agenda - Recommendations –Rec. 4.10: Develop new methods and technologies for capturing and representing information quality and uncertainty –Rec. 4.11: Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment. – Summary Rec: Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process

4 Uncertainty Analysis as Resource to Visual Analytics VA Agenda –Develop new methods and technologies for capturing and representing information quality and uncertainty –Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment. –Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process UA Insight –Probabilistic techniques Elicitation methods Aggregation methods Information-theoretic approaches –Nonprobabilistic alternatives Dempster-Shafer Possibility theory –Uncertainty propagation techniques Analytic Numerical –Risk communication Risk representation Decision-analysis methods

5 MEASURING UNCERTAINTY CLASSICAL METHODS BAYESIAN METHODS NON- PROBABILISTIC METHODS

6 Classical Statistics Focus on Aleatory Uncertainty –random variation inherent in the system Sampling produces confidence intervals Need a sampling model –Generally unavailable for many real-world complex situations Confidence intervals are not probability intervals –Propagation difficulties in even the simplest models

7 Bayesianism de Finetti, Ramsey, Savage (1920s-50s) Subjectivism – Epistemic Probabilities –Probability as a degree of belief Classicists are coin tossers Bayesians are believers –What is the basis for forming probability? “ Probabilities do not exist” –Bruno de Finetti

8 Problems with Bayesianism Because probabilities don’t exist, they have to be created –but how? Bayes’ Theorem Subjectivity is explicit –judgment of evidence Do probabilities really reflect the way we conceive belief? –is probability theory a good theory of evidence? –what are the options?

9 One Option: Dempster-Shafer Theory Withholding belief distinct from disbelief Seahawks or Steelers will win? Set of possibilities: {sea, steel} Probability theory: –Weight of evidence attached to each exclusive possibility – p(sea), p(steel) D-S theory: –Weight of evidence attached to each subset –m(Ø), m(sea), m(steel), m(sea U steel) Allows: m(sea U steel) = 1, all other m=0 –A compelling ignorance

10 Support and Plausibility Probability replaced by two belief measures: –Each calculated from weights of evidence –bel(sea) is the support for proposition ‘sea’ –pl(sea) is the plausibility of ‘sea’ –bel(sea) ≤ pl(sea) –Upper and lower “probabilities” Complete ignorance SDU: bel(sea) = 0, pl(sea) = 1, i.e., complete ignorance on the matter of proposition ‘sea’

11 Complementary Cumulative Belief Functions Complementar y Cumulative Probability

12 Possibility Theory Genesis in fuzzy sets Possibility is an uncertainty measure that mirrors the fuzzy set notion of imprecision

13 The Set of Tall Men

14 Possibility Theory 2-tier belief: possibility and necessity nec(X) ≤ pos(X) Distinctive combinatorial logic –nec(X^Y) = min[nec(X), nec(Y)] –pos(XvY) = max[pos(X), pos(Y)] No conceptual connection to probability –although probability/possibility can co-exist

15 Possibilistic Interpretation of Intelligence Statements (Heuer) Probability Possibility Chances are slight Little chance Better than even Highly likely

16 Experience with Nonprobabilistic Methods Not all good: –Standardization of belief metrics? –Treatment of dependences? –Treatment of conflicting evidence? –Computational demands? –Interpretation of results? –Incorporation into decision process? Plan B: Confront the problems with probabilistic methods

17 Principled Basis for Probability Formulation Analysts uncomfortable producing probabilities –justified discomfort Alternative: –Produce defensible basis for probability formulation based on nonprobabilistic judgment Maximize expression of uncertainty subject to judged constraints Borrow uncertainty metrics from: –statistical mechanics –information theory Entropy = -∑ i p i.ln p i –discrete probability distribution, p i

18 Application of Information- Theoretic Methods Two USNRC programs: –QUEST- SNL Quantitative uncertainty evaluation of source terms –QUASAR – BNL Quantitative uncertainty analysis of severe accident releases Both studies used the same form of input to the same deterministic models –non-probabilistic input expert-generated input parameter uncertainty ranges QUEST:Bounding analysis QUASAR:Information Theory used to generate probability distributions from bounds Probabilistic analysis internal to methodology – no elicitation of probability

19 Information Theory and the Preservation of Uncertainty Uncertainty Bands

20 Uncertainty Analysis as Resource to Visual Analytics VA Agenda –Develop new methods and technologies for capturing and representing information quality and uncertainty –Determine the applicability of confidence assessment in the identification, representation, aggregation, and communication of uncertainties in both the information and the analytical methods used in their assessment. –Develop methods and principles for representing data quality, reliability, and certainty measures throughout the data transformation and analysis process UA Insight –Probabilistic techniques Elicitation methods Aggregation methods Information-theoretic approaches –Nonprobabilistic alternatives Dempster-Shafer Possibility theory –Uncertainty propagation techniques Analytic Numerical –Risk communication Risk representation Decision-analysis methods

21 Merit Criteria for Uncertainty Analysis in Intel Makes the analyst’s job easier Represents strength of evidence intuitively Can reflect dissonant evidence Appropriately propagates uncertainty from analyst to decision-maker Characterizes output uncertainty in a standardized and interpretable way Computationally tractable Promotes insight