1. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Expert-Opinion Elicitation Robert C. Patev North Atlantic Division – Regional Technical Specialist (978) 318-8394

2. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Subjective Estimation Elicitation Process –Background –Expert-Opinion Elicitation (EOE) Process Probability –Axioms of Probability –Medians and Percentiles Training Example Expert-Opinion Elicitation

3. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Subjective Estimation Uses of one or more experts to estimate a probability (qualitative or quantitative) for use in engineering risk analysis –Good for first estimate of probabilities –Quick, cost effective and efficient method –Problems: Not a formal elicitation Usually not well documented Probabilities may not be repeatable or defendable Probabilities may be highly subjective and biased Probabilities have larger uncertainties compared to structured elicitation values

4. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Subjective Estimation How good are we at quantifying subjective estimates? Let us see…..

5. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Subjective Estimation How good are we at quantifying subjective estimates? –Class Example: How may ships passed through the Panama Canal last year? –Give best estimate

6. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Background –Process developed by RAND Corporation in late 1950’s - early 1960’s Delphi Method Scenario Analysis –Effects of thermonuclear war –Civil Defense strategic planning Examine if U.S. population could survive a nuclear attack Expert-Opinion Elicitation

7. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Background –Definition –A formal (protocol), heuristic (through discussion) process of obtaining information or answers to specific questions called issues e.g., failure rates or probabilities, and failure consequences Expert-Opinion Elicitation

8. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Background –EOE is used for preliminary risk evaluation (screening) is not really intended to replace more complex reliability models –EOE has been used by industry and government agencies to develop failure probabilities when there is a lack of failure information Expert-Opinion Elicitation

9. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Drawbacks –Subjective process Not consensus building Inherently contains bias and dominance –Difficult to process result to determine reliability or hazard rates Assumptions need to be made Current Usage in USACE –Supplement to other models Calculate reliability (not for critical components) Event tree probabilities Used in consultation with HQUSACE Expert-Opinion Elicitation

10. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions EOE Process –Participants Experts Observers Listeners Technical Integrator and Facilitator Peer Reviewers –ITR process and results Expert-Opinion Elicitation

11. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions EOE Process –Identification and Selection of Experts Strong relevant expertise Familiarity and knowledge with issues Willingness to act as impartial evaluators Willingness to participate, prepare, and provide needed input Strong communication skills, interpersonal skills, and ability to generalize Expert-Opinion Elicitation

12. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions EOE Process –Inform experts of issues “Read ahead” materials Site visits –Train experts –Elicitation First opinion Discussion among experts Second opinion Expert-Opinion Elicitation

13. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Probability –General expressions Percent (1% probability of failure) Fraction (1/100) Relative frequency (1 out of 1000) –Axioms of Probability 0 < P f < 1 Sum of probabilities over all possible outcomes must equal 1. –This assume events are independent. Expert-Opinion Elicitation

14. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Statistics –Median e.g., Median income, median age Rank value For odd n, value with rank of (n+1)/2 For even n, average of value with rank n/2 or (n/2) + 1 Used to limit extreme values –Average Sum of Xi divided by sample size Expert-Opinion Elicitation

15. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Median Vs. Average Sample 1 –100 –200 –300 –400 Median = 200 Average = 220 Sample 2 –100 –200 –300 –2000 Median = 200 Average = 540

16. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Percentiles –A p-percentile value (Xp) based on a sample is the value of the parameter such that p% of the data is less than or equal to Xp e.g., The median is the 50th percentile Expert-Opinion Elicitation

17. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Chickamauga Lock and Dam

18. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Chickamauga Lock and Dam

19. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Chickamauga Lock and Dam

20. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Chickamauga Lock and Dam

21. “ Building Strong “ Delivering Integrated, Sustainable, Water Resources Solutions Class Example –Six experts required –Unknown issue given to experts Define assumptions of issue Elicit first values First results Expert Discussion Elicit second values Show final elicitation results Expert-Opinion Elicitation