1. 1 Use of SEM programs to precisely measure scale reliability Yutaka Kano and Yukari Azuma Osaka University IMPS2001, July 15-19,2001 Osaka, Japan

2. 2 Reliability measure for Reliability with possibly correlated errors α coefficient

3. 3 An example α 0.69 0.74 0.78 ρ ’ 0.69 0.64 0.60

4. 4 From the example Coefficient alpha can be distorted seriously by error correlations e.g. Green-Hershberger (2000), Raykov (2001) In the case, ρ ’ has to be used to correctly figure out the reliability

5. 5 Problem How can one identify error correlations? The factor model allowing for (fully) correlated errors is not identifiable, because it contains too many parameters A trivial solution would be It does not work because

6. 6 LM approach Start from the factor model with no error correlation Perform the LM test for releasing a zero covariance between errors using a SEM program EQS can perform it most easily and most accurately

7. 7 Real data analysis A questionnaire on perception on physical exercise n=653, p=15, one-factor model The data were collected by Dr. Oka (Waseda U.)

8. 8 Result_1 Best fitted model, with 7 correlated errors χ2=250.375(df=83) (n=653) GFI=0.950, CFI=0.952, RMSEA=0.056

9. 9 Result_2 Estimates of reliability α = 0.90 ρ ’ = 0.90 by ρ ’ = 0.87 by LM test Note that

10. 10 Search for variables Even though one factor model is fitted well, inclusion of a variable with small true variance can reduce reliability There is no convenient way to select variables for the composite scale to have maximum reliability

11. 11 Mathematically … It is complicated It will become more complicated if error correlations are allowed

12. 12 New program A new program is being developed which gives a list of reliability estimates for each factor; gives a list of predicted reliability estimates when one variable is removed Error correlations are allowed

13. 13 Flowchart Decide composite scale items DATA Factor analysis Well fit? Free some error covariances to get good fit Print reliability No Yes End

14. 14 Example, continued ρ ’ = 0.87 with 15 variables

15. 15 Scale developer

16. 16 If V13 is removed, then …

17. 17 Results For one-factor model with uncorrelated errors, the variable with the smallest factor loading is least favorable. If there is a variable whose deletion improves reliability, then this is the variable. For one-factor model with correlated errors, the variable with the smallest factor loading is not always least favorable. While deletion of the variable does not improve reliability, there may be other variables to be deleted to improve reliability. The example here is the case.

18. 18 Summary Correlated errors invalidate the coefficient alpha and traditional one-factor based reliability. LM test is useful to find error correlations. Magnitude of factor loadings does not necessarily provide accurate information on indicator selection when correlated errors exist. The forthcoming Web-based program will help reliability analysis.