A Method for the More Accurate Measurement and Communication of Model Error Scott Fortmann-Roe University of California, Berkeley

1) More accurate assessment of prediction error Predictions Inferences 2) More accurate models 3) More accurate measures of significance 4) Altered inferences and conclusions

Issues with Current Approaches

Measure R 2, p- value, AIC AccuracyAccessibilityAdaptability

Measure Accuracy (R 2 ) AccessibilityAdaptability

House Area House Price

Measure Accuracy Accessibility (p-values) Adaptability

[Given a p-value from an experiment] you have found the probability of the null hypothesis being true. “

Measure AccuracyAccessibility Adaptability (AIC, BIC, …)

The Method: A 3

Does X significantly affect Y? Does the inclusion of X in a model increase our ability to predict Y?

High-Level Statistical Overview  Wraps around any predictive algorithm  Linear Regression, Logistic Regression, Random Forests, …  Cross-validation is used to obtain accurate measure of error  Exact test is used to obtain accurate p-values  No parametric assumptions (other than independence between observations)  (Even independence may be violated if compensated for)

Applications

Housing Market Predicting housing price based on house and market attributes Harrison D, Rubinfeld DL (1978) Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management 5: 81–102.

Coefficient Std. Error t-Valuep-Value (Intercept) AGE ROOMS < 0.01 NOX < 0.01 PUPIL/ TEACHER < 0.01 HIGHWAY Adjusted R 2 : 0.60; p-Value < 0.01

CoefficientCrVa R 2 p-Value -Full Model %< 0.01 (Intercept) %0.39 AGE %0.22 ROOMS %< 0.01 NOX %< 0.01 PUPIL/ TEACHER %< 0.01 HIGHWAY %1.00 A 3 : Linear Model

CrVa R 2 p-Value -Full Model-74.3 %< 0.01 AGE- 1.5 %0.01 ROOMS %< 0.01 NOX+ 6.3 %< 0.01 PUPIL/ TEACHER %< 0.01 HIGHWAY- 2.6 %0.03 A 3 : Random Forest Model

Linear Regression Random Forest Support Vector Machines CrVa R Significant at p = 0.05 ROOMS NOX PUPIL/TE ACHER AGE ROOMS NOX PUPIL/TE ACHER HIGHWAY AGE ROOMS NOX PUPIL/TE ACHER Not Significant at p = 0.05 AGE HIGHWAY

Environmental Productivity Measure utility of an ecosystem based on different physical attributes Maestre FT, Quero JL, Gotelli NJ, Escudero A, Ochoa V, et al. (2012) Plant Species Richness and Ecosystem Multifunctionality in Global Drylands. Science 335: 214–218.

CoefficientStd. Errort-Valuep-Value (Intercept) < 0.01 SR SLO < 0.01 SAC < 0.01 C C C C < 0.01 LAT LONG < 0.01 ELE < 0.01 Adjusted R 2 =0.56; p-Value < 0.01

CoefficientCrVa R 2 p-Value -Full Model %< 0.01 (Intercept) %< 0.01 SR %0.01 SLO %0.01 SAC %< 0.01 C %0.91 C %0.15 C %0.28 C %< 0.01 LAT %0.09 LONG %< 0.01 ELE %< 0.01 A 3 : Linear Model

CrVa R 2 p-Value -Full Model-68.3 %< 0.01 SR+ 1.2 %< 0.01 SLO- 1.3 %0.95 SAC+ 4.0 %< 0.01 C %< 0.01 C %0.02 C %0.16 C %< 0.01 LAT+ 0.5 %< 0.01 LONG+ 0.2 %0.02 ELE+ 0.4 %0.02 A 3 : Random Forest Model

Applications Recap  Explained an additional 15-16% of the squared error  Significantly altered inferences and conclusions about the underlying systems

Summary

MethodAccuracyAccessibilityAdaptability R2R2 ★☆☆ ★★★ Adjusted R 2 ★★☆★★★★☆☆ p-Values ★★★★★☆ AIC, BIC and Information Theoretic Techniques ★★★★☆☆★★☆ A3A3 ★★★

Questions….