Expert Judgment short course, NIA, 15,16 April, 2008 UNCERTAINTY AMBIGUITY 5: Utilities In Stakeholders Population Roger Cooke Resources for the Future Dept. Math, Delft Univ. of Technology April 15,16 2008 INDECISION

UNCERTAINTY Do measurements, Quantify uncertainty AMBIGUITY Define concepts, Domain of application INDECISION Quantify utilities, preferences

UNCERTAINTY ‘What is’: Possible to answer AMBIGUITY ‘What means’: Easy to answer INDECISION ‘What’s best’: ??

Fundamental Theorem of Decision Theory For Rational Preference There is a UNIQUE probability P which represents degree of belief: DegBel(F) > DegBel(US)  P(F) > P(US) AND a Utility function, unique up to 0 and 1, that represents values: L(F) > L(US)  Exp’d Utility (L(F)) > Exp’d Utility (L(US)) BUT…. Ambiguity Indecision Uncertainty 5

There is no Updating utilities on observations Convergence of utilities via Observations Empirical control on Utilities Community of ‘Utility Experts’ Rational consensus on Utilities Ambiguity Indecision Uncertainty 5

Condorcet’s Paradox of Majority Preference 1/3 prefer Mozart > Hayden > Bach 1/3 prefer Hayden > Bach > Mozart 1/3 prefer Bach > Mozart > Hayden THEN 2/3’s prefer Bach > Mozart Mozart > Hayden Hayden > Bach Ambiguity Indecision Uncertainty 5

Funny little theorem Let pij = prob{i preferred to j}, i,j = 1,2,3. Theorem:There exists a distribution of rankings 3! Which reproduces {pij}, i.e. pij = Prob{ all rankings with i > j} If and only if pij + pjk + pki  2, for all unequal i,j,k  {1,2,3}. Proof (15 min) Discrete_choice_w_PI.pdf Little is known for more than 3 alternatives Ambiguity Indecision Uncertainty 5

Random Utility Theory What can we do? Each (rational) stakeholder has a utility function over alternatives  characterize population as distribution over utility functions Ambiguity Indecision Uncertainty 5

Other approaches try to build ONE utility function MCDM MAUT AHP Etc etc Ambiguity Indecision Uncertainty 5

For alternatives A,B,C, D… answer questions like: ‘how close/far are values of A, B in distributional sense?’ ‘Is someone who prefers A>B more likely to prefer C>D? than someone with A<B” Ambiguity Indecision Uncertainty 5

Paired Comparison Demo Case: Energy Policies 1: Tax@pump: 1$ per gallon gasoline surcharge, to be used for research in renewables 2:Tax Break: (a) No sales tax for purchase of new hybrid or electric car; (b)First car owners can deduct purchase cost from their income tax; (c) No sales tax on bio-diesel or ethanol (c) Tax credits for energy efficiency home improvements (insulation, double glass windows, solar panels) 3. Vehicle Tax: Annual road tax 1$ per lb on all light duty vehicles, no tax rebate for driving to work or parking, to be used for research in fuel efficient vehicles and bio fuels. 4. CO2 cap CO2 emissions cap on electricity generation. 5. Subsidies for clean coal Give subsidies for clean coal with carbon sequestration to make coal competitive with natural gas. 6. Do Nothing Ambiguity Indecision Uncertainty 5

Enter Data in UNIBALANCE Help_for_Unibalance.doc Reference values can fix scale of these models Bradley-Terry, ratio scale Thurstone, affine scale Ambiguity Indecision Uncertainty 5

Check each Stakeholders’ consistency Paired comparisons and group preference.doc Help_for_Unibalance.doc Ambiguity Indecision Uncertainty 5 p-value for rejecting the hypothesis that pairwise preferences are at random, based on the nr of circular triads

Check that group is statistically different from ‘random stakeholders’ Ambiguity Indecision Uncertainty 5 Variance in stakeholder-averaged ranks How much stakeholders pairwise agree

Paired Comparison Data Analysis Is each stakholder’s preference non-random? Is the agreement between stakholders non-random? Coefficient of agreement Coefficient of concordance Ambiguity Indecision Uncertainty 5

Three Ways to Analyse Paired Comparison Data Thurstone (probit) models Bradley-Terry (logit) models Probability Inversion Ambiguity Indecision Uncertainty 5

Thurstone model (1927) aka probit model MODEL C: Values are distributed as indep. normals  = 1, over stakeholders: Find relative placement that reproduces PC preferences Clean Coal Tax@Pump Do Nothing Ambiguity Indecision Uncertainty 5 MODEL B:  = CONST, |I - J |<< I

Bradley Terry model (1952) aka logit model %{ i > j} ~ value(i) / [value(i) + value(j)] Likelihood of data  pairs(vi / (vi+ vj))#{i>j} (vj / (vi+ vj))#{j>i} Find maximum likelihood estimate of v’s Ambiguity Indecision Uncertainty 5

Are Choices Independent? Ambiguity Indecision Uncertainty 5 5% prefer a Mercedes 300 to a Panda >  Given that someone prefers Merc>Panda, what about BMW > Corsa? Still 5%???? >

Probabilisic Inversion Each (rational) person/stakeholder/consumer has a utility function The population = distribution over utility fn’s Start from non-informative distribution and adapt to find dist’n over utilities which matches paired comparison data Ambiguity Indecision Uncertainty 5

Standardize all utility functions: heaven = 1, hell = 0, non-informative (UNIGRAPH) Ambiguity Indecision Uncertainty 5 One random stakeholder 1000 random sakeholders Conditionalization doesn’t change others Marginal utilities all the same

Re-weight sample to impose PC constraints Ambiguity Indecision Uncertainty 5 Re-weight sample to impose PC constraints Conditioning changes others

Probabilistic inversion Generic_Prob_Inversion Probabilistic inversion Generic_Prob_Inversion.pdf; Paired comparisons and group preference.doc EJcoursenotes-PI.doc G: Rn → Rm x1,..xn  G-1(y1,..ym) if G(x1,..xn) = (y1,..ym) or; C  Rm; x1,..xn  G-1(C) if G(x1,..xn)  C Now let X,, Y be random vectors, C  {random vectors on Rm} X1,..Xn  G-1(Y1,..Ym) if G(X1,..Xn) ~ (Y1,..Ym) X1,..Xn  G-1(C) if G(X1,..Xn)  C Here, U = (U1, .. U6) G(U) ~ (1{1>2}, ... 1{5 > 6}) Ambiguity Indecision Uncertainty 5

What do we know? Iterative Proportional Fitting algorithm (IPF, Kruithof 1937) fits each constraint sequentially. If feasible, it converges to max. likelihood dist’n relative to start dist’n (fast) PARFUM algorithm fits each constraint singly and averages. It converges to a solution if feasible, and to a ‘minimally unlikely’ dist’n else. (slow) Ambiguity Indecision Uncertainty 5

Ambiguity Indecision Uncertainty 5 Mazzuchi: Shorting failure (IPF: 16,000 samples, 1000 iterations RESS-Mazzuchi_Linzey_Brunin.pdf) Bad Fit, IPF not converging, high correlations and bumpy densities 72% of experts preferred 1>3, 41% in IPF sample

Mazzuchi: Shorting failure (PARFUM: 16,000 samples, 1000 iterations f) Ambiguity Indecision Uncertainty 5 PARFUM converges to least ugly misfit, smoother densities, lower correlations

UNISENS can perform standard sensitivity analyses Ambiguity Indecision Uncertainty 5

Gives regression lines: how values of alternatives covary Ambiguity Indecision Uncertainty 5

Regress average scores on explanatory variables Ambiguity Indecision Uncertainty 5

More general discrete choice Given 30 scenarios, rank top 5 Do PI directly on explanatory vbls, eg: Vi = j Bj Xj,i Find distribution over B’s to represent P(Vi > Vj) Ambiguity Indecision Uncertainty 5

Fake data for decadal survey ..\EJ-Programs\Unibalance.exe Ambiguity Indecision Uncertainty 5