Assessing Hypothesis Testing Fit Indices Kline Chapter 8 (Stop at 210) Beaujean Appendix A

Fit Indices It’s complicated yo. There are a lot of them. They do not imply that you are perfectly right. They have guidelines but they are not perfect either. People misuse them. Etc.

Fit Indices Limitations: Fit statistics indicate an overall/average model fit. That means there can be bad sections, but the overall fit is good. No one magical number/summary. They do not tell you where a misspecification occurs.

Fit Indices Limitations Do not tell you the predictive value of a model. Do not tell you if it’s theoretically meaningful.

Fit Indices What size? I need a rule?! Everyone cites Hu and Bentler (1999) for the golden standards. Same problem that Cohen had (we love rules). So when the fit is messy, cite Kline (page 197) as reasons that’s not a bad thing This section is an interesting read, especially if you have trouble publishing, but not crucial to your understanding of fit indices

Fit Indices Model test statistic – examines if the reproduced correlation matrix matches the sample correlation matrix Sometimes called “badness of fit” Want these to be small

Fit Indices Traditional NHST = reject-support context You reject the null to show your research hypothesis is correct. SEM Hyp Testing = accept-support context You do not reject the null showing that your model is consistent with the population

Model Test Statistic Chi-square Formula = (N-1)FML FML = is the minimum fit function in ML estimation P values are based on df for your model and a chi-square distribution You want this to be nonsignificant. But this is a catch 22!

Model Test Statistic Chi-square is biased by Multivariate non-normality Correlation size – bigger correlations can be bad for you (harder to estimate all that variance) Unique variance Sample size

Model Test Statistic Everyone reports chi-square, but people tend to ignore significant values (I’m sort of eh on his YOU MUST PAY ATTN OR DIE talk in this section)

Model Test Statistic Normed chi-square or (X2/df) – this used to be widely reported and used The criterion was < 3.00 were good models Now most people have moved away from this procedure (i.e. don’t use this one). Also considered an absolute fit index.

Model Test Statistic Since X2 is biased by a bunch of things, you can use a robust estimator to help fix that bias. Satorra-Bentler Yuan-Bentler See page 157 for all the different robust options

Fit Indices Many of the statistics described next are compared to the following model types: Independence – a model assuming no relationships between the variables (i.e. parameters are not significant) Saturated – a model assuming all parameters exist (i.e. df = 0).

Fit Indices Both types of statistical inferences have their problems … and especially in SEM because it is easy to find statistics that you would normally reject, even with good model fit. Tends to be too black and white (reject or not to reject!)

Fit Indices Alternative Fit Indices Not traditionally a dichotomous yes-no decision Do not distinguish between sampling error and evidence against the model

Fit Indices Alternative Fit Indices Absolute Fit Indices Incremental Fit Indices Parsimony-adjusted Index Predictive Fit Indices

Fit Indices Absolute fit indices Proportion of the covariance matrix explained by the model You can think about these as sort of R2 Want these values be high

GFI Do not use this sucker unless you want to get a nasty review. GFI, AGFI, PGFI Lots of research showing it’s positively biased

SRMR Standardized root mean (square) residual Want small values Excellent < .06 (not a typo different than book) Good < .08 Acceptable < .10 Eeek > .10

Fit Indices Incremental Fit Indices Also known as comparative fit indices Compared to the improvement over the independence model (remember that’s the one with no relationships between the variables) Not necessarily the best indices

CFI Comparative Fit Index Values are 0 to 1 (sometimes you’ll get slightly over 1, usually indicates something is wrong) Want high values Excellent >.95 Good > .90 Blah < .90 All the following FIs have the same cut off rules

NFI Normed Fit Index A variation of the CFI, as it was said to underestimate for small samples 1 – (X2baseline / X2model)

IFI Incremental Fit Index Also known as Bollen’s Non-normed fit index Modified NFI that doesn’t depend on sample size so much. (X2baseline - X2model) / (X2baseline - dfmodel)

RFI Relative Fit Index Also known as Bollen’s Normed Fit Index (X2model/dfmodel) / (X2baseline/dfbaseline)

TLI Tucker Lewis Index Also known as the Bentler-Bonet Non-Normed Fit Index ( (X2model/dfmodel) - (X2baseline/dfbaseline) ) / ( (X2baseline/dfbaseline) – 1)

Fit Indices Parsimony-adjusted index These include penalties for model complexity Normally more paths = better fit. These will have smaller values for simpler models

RMSEA Root mean squared error of approximation Parsimony-adjusted index Want small values Excellent < .06 (not a typo different than book) Good < .08 Acceptable < .10 Eeek > .10 Report Confidence Interval!

Pclose Tests if the RMSEA is in the excellent range You want p > .50 to show that there is a high probability that RMSEA is effectively zero

Fit Indices Predictive fix indices Estimate model fit in a hypothetical replication of the study with the same sample size randomly drawn from the population Not always used Often used for model comparisons, see below. Often also consider parsimony adjusted indices.

Model Comparisons Let’s say you want to adjust your model You can compare the adjusted model to the original model to determine if the adjustment is better Let’s say you want to compare two different models You can compare their fits to see which is better

Model Comparisons Nested models If you can create one model from another by the addition or subtraction of parameters, then it is nested Model A is said to be nested within Model B, if Model B is a more complicated version of Model A. For example, a one-factor model is nested within a two-factor as a one-factor model can be viewed as a two-factor model in which the correlation between factors is perfect).

Nested Models Chi-square difference test | Subtract Model 1 X2 – Model 2 X2| Subtract Model 1 df – Model 2 df Use a chi-square table to look up p < .05 for difference in df See if the first step is greater than that value If yes, you say the model with the lower chi-square is better If no, you say they are the same and go with the simpler model

Nested Models CFI difference test Subtract CFI model 1 – CFI model 2 If the change is more than .01, then the models are considered different This version is not biased by sample size issues with chi-square.

Nested Models So how can I tell what to change? NOTE: JUST CHANGE ONE THING AT A TIME! Use modification indices! They tell you what the chi-square change would be if you add the path suggested. Based on X2(1) – called a Lagrange Multiplier Remember that p < .05 = 3.84

Nested Models Can be tested with lavaan using the anova() function. More on this in the multigroup section

Non-Nested Models AIC – Akaike Information Criterion BIC – Bayesian Information Criterion SABIC – Sample size Adjusted Information Criterion All of these penalize you for having more complex models. If all other things are equal, it is biased to the simpler model.

Non-Nested Models All of the ICs are how much the sample will cross validate in the future You want them to be small, so you pick the smallest one of the two models (how different?)

Non-Nested Models ECVI – expected cross validation index Fmin + (2t / ( n – p – 2) ) T number of parameters estimated P number of squares Again, you want small values, so you pick the model with the smallest ECVI

OMG! So what to do? Mainly people report: X2(df), RMSEA, SRMR, CFI Determine the type of model change to use the right model comparison statistic

Example Two models Let’s make the models And compare them with their fit indices Are they nested or not? See handout on blackboard.