Presentation is loading. Please wait.

Presentation is loading. Please wait.

Experrt Judgment short course, NIA 15,16 April 2008

Published byὙπατια Καραβίας Modified over 5 years ago

Similar presentations

Presentation on theme: "Experrt Judgment short course, NIA 15,16 April 2008"— Presentation transcript:

1 Experrt Judgment short course, NIA 15,16 April 2008
UNCERTAINTY 3: Bumper Stickers for Expert Judgment AMBIGUITY Roger Cooke Resources for the Future Dept. Math, Delft Univ. of Technology April 15, INDECISION

2 1. Expert Judgment is NOT Knowledge
Scientific method – NOT EJ methods - produces agreement among experts EJ is for quantifying ....not removing..... uncertainty. Ambiguity Indecision Uncertainty 3

3 2. Experts CAN quantify uncertainty as subjective probability
Ambiguity Indecision Uncertainty 3 TU DELFT Expert Judgment database 45 applications (anno 2005): # experts variables elicitations Nuclear applications 98 2,203 20,461 Chemical & gas industry 56 403 4,491 Groundwater / water pollution / dike ring / barriers 49 212 3,714 Aerospace sector / space debris /aviation 51 161 1,149 Occupational sector: ladders / buildings (thermal physics) 13 70 800 Health: bovine / chicken (Campylobacter) / SARS 46 240 2,979 Banking: options / rent / operational risk 24 119 4,328 Volcanoes / dams 231 673 29079 Rest group 19 762 TOTAL 521 3688 67001

4 Including 6700 calibration variables
Study Variables of interest calibration variables Dispersion Far-field dispersion coeff’s Near-field tracer exper’ts Environmental transport Transfer coeff’s Cumulative concentrations Dose-response models Human dose response Animal dose response Investment pricing Quarterly rates Weekly rates PM2.5 Long term mortality rates Short term mortality ratio’s Ambiguity Indecision Uncertainty 3

5 Calibration questions for PM2.5 RESS-PM25.pdf
In London 2000, weekly average PM10 was 18.4 μg/m3. What is the ratio: # non-accidental deaths in the week with the highest average PM10 concentration (33.4 μg/m3)  Weekly average # non-accidental deaths. 5% :_______ 25%:_______ 50% :_______ 75%:________95%:________ Ambiguity Indecision Uncertainty 3

6 3. Experts confidence does NOT predict performance
Very informative assessors may be statistically least accurate PM25-Range-graphs.doc Ambiguity Indecision Uncertainty 3

7 4. Experts are sometimes well calibrated
AMS-OPTION-TRADERS-RANGE-GRAPHS.doc realestate-range graphs.doc Sometimes not GL-invasive-species-range-graphs.doc Ambiguity Indecision Uncertainty 3

8 5. Experts sometimes agree
Dispersion-USNRC-EU-RANGE-GRAPHS.doc And sometimes don’t Campy-range-graphs.doc Earlyhealth-USNRC-EU-Range-graphs.doc Ambiguity Indecision Uncertainty 3

9 6. Classical model usually works, but not always
All experts, and all combinations can be bad: Soil-animal-USNRC-EU-range-graphs.doc Ambiguity Indecision Uncertainty 3

10 7. We CAN do better than equal weighting RESS-TUDdatabase.pdf
Ambiguity Indecision Uncertainty 3

11 Ambiguity Indecision Uncertainty 3

12 Out-of-sample Validation RESS_response2comments.pdf
Ambiguity Indecision Uncertainty 3 13 studies with  14 seed vbls, split, initialize on one half, predict other half

13 8. Citation-based (social network) weights do NOT perform better RESS-SocNet&Likelwgts.pdf
Ambiguity Indecision Uncertainty 3

14 9. Experts like performance assessment
Ask them Aspinall_mvo_exerpts.pdf, Aspinall et al Geol Soc _.pdf , Aspinall & Cooke PSAM4 3-9.pdf, SparksAspinall_VolcanicActivity.pdf Separate scientific assessment of uncertainty from decision making Ambiguity Indecision Uncertainty 3

15 Experrt Judgment short course, NIA 15,16 April 2008
Ambiguity Indecision Uncertainty 3 “ In the first few weeks of the Montserrat crisis there was perhaps, at times, some unwarranted scientific dogmatism about what might or might not happen at the volcano, especially in terms of it turning magmatic and explosive. The confounding effects of these diverging, categorical stances were then compounded for a short while by an overall diminution in communication between scientists and the various civil authorities. The result was a dip in the confidence of the authorities in the MVO team and, with it, some loss of public credibility; this was not fully restored until later, when a consensual approach was achieved. “ Aspinall et al The Montserrat Volcano Observatory: its evolution, organization, rôle and activities.

16 10. The choice is NOT whether to use EJ; but: do it well or do it badly? “NASA must always strive for the highest level of accomplishment, to exceed the expectations of the Nation, and to do what is right...”(Return to Flight task group p.14) Ambiguity Indecision Uncertainty 3

17 Lets have another break
Ambiguity Indecision Uncertainty 3

Download ppt "Experrt Judgment short course, NIA 15,16 April 2008"

Similar presentations

What can Statistics do for me? Marian Scott Dept of Statistics, University of Glasgow Statistics course, September 2007.

What can Statistics do for me? Marian Scott Dept of Statistics, University of Glasgow Statistics course, September 2007.
General Statistics Ch En 475 Unit Operations. Quantifying variables (i.e. answering a question with a number) 1. Directly measure the variable. - referred.

General Statistics Ch En 475 Unit Operations. Quantifying variables (i.e. answering a question with a number) 1. Directly measure the variable. - referred.
Basic Optimization Problem Notes for AGEC 641 Bruce McCarl Regents Professor of Agricultural Economics Texas A&M University Spring 2005.

Basic Optimization Problem Notes for AGEC 641 Bruce McCarl Regents Professor of Agricultural Economics Texas A&M University Spring 2005.
Environmental Risk Analysis

Environmental Risk Analysis
Environmental Health: Risk Assessment and Risk Management Michael Brody, PhD Senior Environmental Scientist 2 December 2011 Almaty, Kazakhstan.

Environmental Health: Risk Assessment and Risk Management Michael Brody, PhD Senior Environmental Scientist 2 December 2011 Almaty, Kazakhstan.
WELCOME THE 12 TH ANNUAL MARCES/MSDE CONFERENCE: Value Added Modeling and Growth Modeling with Particular Application to Teacher and School Effectiveness.

WELCOME THE 12 TH ANNUAL MARCES/MSDE CONFERENCE: Value Added Modeling and Growth Modeling with Particular Application to Teacher and School Effectiveness.
Task Force on Health Recent results - Particulate matter Michal Krzyzanowski TFH Chair Head, Bonn Office European Centre for Environment and Health WHO.

Task Force on Health Recent results - Particulate matter Michal Krzyzanowski TFH Chair Head, Bonn Office European Centre for Environment and Health WHO.
General Statistics Ch En 475 Unit Operations. Quantifying variables (i.e. answering a question with a number) Each has some error or uncertainty.

General Statistics Ch En 475 Unit Operations. Quantifying variables (i.e. answering a question with a number) Each has some error or uncertainty.
U.S. EPA DISCLAIMER EPA strongly cautions that these study results should not be used to draw conclusions about local exposure concentrations or risk.

U.S. EPA DISCLAIMER EPA strongly cautions that these study results should not be used to draw conclusions about local exposure concentrations or risk.
Introduction to Risk Analysis in Healthcare Farrokh Alemi Ph.D. Professor of Health Administration and Policy College of Health and Human Services, George.

Introduction to Risk Analysis in Healthcare Farrokh Alemi Ph.D. Professor of Health Administration and Policy College of Health and Human Services, George.
Watershed Management Assessment Through Modeling: SALT and CEAP Dr. Claire Baffaut Water Quality Short Course Boone County Extension Office April 12, 2007.

Watershed Management Assessment Through Modeling: SALT and CEAP Dr. Claire Baffaut Water Quality Short Course Boone County Extension Office April 12, 2007.
Stephen G. CECCHETTI Kermit L. SCHOENHOLTZ Understanding Risk Copyright © 2011 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill/Irwin.

Stephen G. CECCHETTI Kermit L. SCHOENHOLTZ Understanding Risk Copyright © 2011 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill/Irwin.
Stakeholder Preference Modeling with Probabilistic Inversion Roger M. Cooke Resources for the Future Dept Math, TU Delft June 16, 2011.

Stakeholder Preference Modeling with Probabilistic Inversion Roger M. Cooke Resources for the Future Dept Math, TU Delft June 16, 2011.
TEACHING STATISTICS CONCEPTS THROUGH STOCK MARKET CONTEXTS Larry Weldon Simon Fraser University.

TEACHING STATISTICS CONCEPTS THROUGH STOCK MARKET CONTEXTS Larry Weldon Simon Fraser University.
Environmental Science Chapter 2 – Scientific Tools Test Review

Environmental Science Chapter 2 – Scientific Tools Test Review
Copernicus Institute Universiteit Utrecht Taking uncertainty on board in decision making The example of adaptation to climate change.

Copernicus Institute Universiteit Utrecht Taking uncertainty on board in decision making The example of adaptation to climate change.
Basic principles on Risk Communication Cristiana Salvi Information and Outreach Unit Special Programme on Health and Environment WHO Regional Office for.

Basic principles on Risk Communication Cristiana Salvi Information and Outreach Unit Special Programme on Health and Environment WHO Regional Office for.
Www.inffer.org Benefit: Cost Ratio David Pannell School of Agricultural and Resource Economics University of Western Australia.

Benefit: Cost Ratio David Pannell School of Agricultural and Resource Economics University of Western Australia.
Part 1d: Exposure Assessment and Modeling Thomas Robins, MD, MPH.

Part 1d: Exposure Assessment and Modeling Thomas Robins, MD, MPH.
RISK DUE TO AIR POLLUTANTS

RISK DUE TO AIR POLLUTANTS

Similar presentations

Ads by Google