1. Model Selection

2. Agenda Myung, Pitt, & Kim Olsson, Wennerholm, & Lyxzen

3. What is a Model? p(t) = w 1 exp(-w 2 t) w 1 = 1.145 w 2 = 0.156

4. What is a Model? p(t) = w 1 exp(-w2 t) w 1 1.145 w 2 = 0.156 80 60 40 20 0 100

5. What is a Model? p(t) = w 1 exp(-w2 t) w 1 1.145 w 2 = 0.156 80 60 40 20 0 100

6. What is a Model? A model is a parametric family of probability distributions. –Parametric because the distributions depend on the parameters. –Distributions because they are stochastic models, not deterministic.

7. Likelihood x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x = data 80 60 40 20 0 100

8. Likelihood x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x L(w 1,w 2 | x) =  f(x i | w 1,w 2 ) ln L(w 1,w 2 | x) =  ln f(x i | w 1,w 2 ) 80 60 40 20 0 100

9. Likelihood x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x L(w 1,w 2 | x) =  f(x i | w 1,w 2 ) ln L(w 1,w 2 | x) =  ln f(x i | w 1,w 2 ) 80 60 40 20 0 100

10. Likelihood The w that maximize ln L(w 1,w 2 | x) =  ln f(x i | w 1,w 2 ) are the maximum likelihood parameter estimates, w 1 * & w 2 *.

11. Falsifiability (Saturation) Old rule of thumb: A model is falsifiable if the number of parameters are less than the number of data points. New rule of thumb: A model is falsifiable iff the rank of the Jacobian is less than the number of data points. –A model is testable if the probability that a model’s predictions are right by chance is 0. –Holds under certain smoothness assumptions.

12. Generalizability Generalizability (as defined by Myung) is the ability to generalize to new data from the same probability distribution. Data are corrupted by random noise & so goodness of fit reflects the models ability to capture regularities and fit noise.

13. Generalizability Goodness of fit = Fit to regularity (Generalizability) + Fit to noise (Overfitting)

14. Generalizability Model ComplexityLowHigh Model Fit Poor Good Goodness of fit Generalizability Overfitting

15. Generalizability Note that the more complex a model is, the more it overfits. Model ComplexityLowHigh Model Fit Poor Good Goodness of fit Generalizability Overfitting

16. Generalizability ModelM1 M2 (TRUE) M3M4 y = w 1 x+e y = ln(x + w 1 ) + e y = w 1 ln(x + w 2 ) + w 3 + e y = w 1 x + w 2 x 2 + w 3 + e Training BOF2.141.851.711.62 Testing BOF2.292.056.483.44

17. Generalizability A good fit can be achieved simply because a model is more flexible. A good fit is necessary, but not sufficient for capturing underlying processes. A good fit qualifies the model as a candidate for further consideration.

18. Generalizability The key for many techniques is to find the model that fits future data best, not necessarily the “true” model. –There is rarely enough data to uniquely identify the true model. –Even if there were, it will probably not be one of the models under consideration. This is not to say that we don’t want to find the “true” model (if there is one).

19. Model Selection The quantity of interest is the lack of generalizability of a model. Essentially: Goodness of fit = Fit to regularity (Generalizability) + Fit to noise (Overfitting) Generalizability = GOF - Overfitting Generalizability = GOF - Complexity So, -Generalizability = -GOF + Complexity

20. AIC Akaike Information Criterion (AIC) AIC is a lack of generalizability measure, so big is bad.

21. AIC AIC = -2 ln L(w * |y) + 2K –y are the data –w * are the MLE estimates –K is the number of model parameters

22. AIC AIC = -2 ln L(w * |y) + 2K Badness of fit: Decreases with parameters. Penalty for complexity: Increases with parameters.

23. AIC AIC measures complexity via number of parameters. Functional form is not considered. 1 parameter2 parameters

24. Model Distance CloserFarther “True” Model

25. AIC AIC selects a model from a set of models that on average minimizes the distance between the model and the “True” model. AIC does not depend on knowing the true model. Given certain conditions, K corrects a statistical bias in estimating this distance.

26. Cross Validation (2-Sample) Data Calibration Model Find w * Validation Use w * CVI

27. Cross Validation Easy to use. Sensitive to functional form of model. Not as theoretically grounded as other methods such as AIC.

28. Cross Validation & AIC In single sample CV, the CVI is estimated from the calibration sample. Where L s is the likelihood of the saturated model with 0 df.

29. Minimum Descriptive Length Suppose the following are data described in bits: –0001000100010001000100010001 –0111010011010000101010101011

30. Minimum Descriptive Length The data can be coded as: –0001000100010001000100010001 for i=1:7, disp(‘0001’), end –0111010011010000101010101011 disp(‘0111010011010000101010101011’)

31. Minimum Descriptive Length Regularity in data can be used to compress the data. The more regular the data are, relative to a particular coding method, the simpler the “program”. –The choice of coding method doesn’t matter so much.

32. Minimum Descriptive Length Think of the ‘program’ as a model. The model that best captures the regularities in the data will give the shortest code length. –0001000100010001000100010001 for i=1:7, disp(‘0001’), end –0111010011010000101010101011 disp(‘0111010011010000101010101011’)

33. Minimum Descriptive Length Capturing data regularities will lead to good prediction of future data, i.e. good generalization. By finding the model with the minimum descriptive length, MDL will find the simplest model that predicts the data well.

34. Minimum Descriptive Length Under certain assumptions, MDL is given by: Badness of fit. Penalty for number of parameters. Penalty for functional form. Doesn’t depend on n.

35. Minimum Descriptive Length Based on I, the Fischer Information Matrix. This term tells you how well the model can fit different data sets by tweaking the parameters.

36. Other Selection Criterion Bayesian model selection Bayesian Information Criterion (BIC) Generalization Criterion Monte Carlo Techniques Etc…