1. Taking Uncertainty Into Account: Bias Issues Arising from Uncertainty in Risk Models John A. Major, ASA Guy Carpenter & Company, Inc.

2. n N=20 observations T = sample mean; =1 true mean T = sample mean; =1 true mean n MLE EP curve: l n q-exceedance point (PML, VaR) l l X.01 = 4.605 actual Example: Exponential Distribution

3. Sampling Distribution of T

4. Estimated PDFs

5. Client Questions n What is the 1 in 100-yr PML (1% VaR)? n What is probability of exceeding 4.605? n Can you give me an EP curve to answer these and similar questions? l Does sampling error affect the answer? l Can I get unbiased answers?

6. 3 Kinds of Bias n “dollar” or X-bias: l the average of PML dollar estimates n “probabilistic” or P-bias: l the average true exceedance probability of estimated PML points n “exceedance” or Q-bias: l the average estimated exceedance probability

7. Exponential MLE is X-unbiased

9. n for small q n Expected actual risk is greater than nominal n Uncertainty increases risk! Exponential MLE is P-biased

11. Correcting for P-bias n Predictive distribution l “Prediction interval” in regression n Mix randomness and uncertainty l integrate model pdf over parameter distribution n Exponential model: n Predictive result:

12. Predictive vs. Model Density

13. Which to use? n MLE curve is X-unbiased l no uncertainty adjustment, but... l on average, gets right $ answer n Predictive curve is P-unbiased l “takes uncertainty into account” and... l on average, reflects true exceedance pr n But they disagree... l and it gets worse...

14. n for small q n Expected estimated risk is greater than the true risk (at the specified threshold) n Uncertainty now causes risk to be overstated! Exponential MLE is Q-biased

16. Correcting for Q-bias n Minimum Variance Unbiased Estimator l standard procedure in classical statistics n Rao-Blackwell Theorem l Expectation of unbiased estimator, conditional on sufficient statistic n Exponential model: n MVUE result:

17. MVUE vs. Model Density

18. Paradox n Say we get an estimated T=1 (correct) n MLE says X.01 =4.605, Pr{X>4.605}=1% n Predictive: X.01 =5.179 is p-unbiased l risk is greater than MLE answer because impact of uncertainty n MVUE: Pr{X>4.605}=.69% is q-unbiased l risk is less because MLE tends to overstate exceedance probability

19. How the Paradox Arises

20. Conclusions n Uncertainty induces bias in estimators n Biases operate in different directions l depends on the question being asked n There is no monolithic “fix” for taking uncertainty into account l Predictive distribution fixes p-bias, l while making q-bias worse

21. Recommendations n First: Show modal estimates (MLE etc.) n Second: Show effect of uncertainty l Keep uncertainty distinct from randomness l Sensitivity testing w.r.t. parameters l Confidence intervals on estimators n Third: Adjust for bias only as necessary l Carefully attend to the question asked l Advise that bias adjustment is equivocal