1. Evaluating Multi-item Scales
Ron D. Hays, Ph.D.
UCLA Division of General Internal Medicine/Health Services Research
HS225A 11/02/10, 3-4:50 pm, CHS 1

2. Example Responses to 2-Item ScaleID
Poor
Fair
Good
Very Good
Excellent 01 2 02 1 03 04 05 2

3. Respondents (BMS) 4 11.6 2.9 Items (JMS) 1 0.1 0.1
Cronbach’s Alpha
01 55
02 45
03 42
04 35
05 22
Respondents (BMS)
Items (JMS)
Resp. x Items (EMS)
Total
Source SS MS df
Alpha =
= = 0.62
2.9
2.9 3

4. Computations Respondents SS Item SS Total SS
( )/2 – 372/10 = 11.6
Item SS
( )/5 – 372/10 = 0.1
Total SS
( ) – 372/10 = 16.1
Res. x Item SS= Tot. SS – (Res. SS+Item SS) 4

5. Alpha for Different Numbers of Items and Average Correlation
Average Inter-item Correlation ( r )
Number
of Items (k) .0 .2 .4 .6 .8
1.0
Alphast = k * r
1 + (k -1) * r 5

6. Spearman-Brown Prophecy Formula( )
N • alpha x
alpha = y
1 + (N - 1) * alpha x
N = how much longer scale y is than scale x 6

7. Example Spearman-Brown Calculation
MHI-18
18/32 (0.98)
(1+(18/32 –1)*0.98
= / = 0.96 7

8. Reliability Minimum Standards
0.70 or above (for group comparisons)
0.90 or higher (for individual assessment)
SEM = SD (1- reliability)1/2 8

9. Intraclass Correlation and Reliability
Model
Reliability
Intraclass Correlation
One-way
Two-way fixed
Two-way random
BMS = Between Ratee Mean Square
WMS = Within Mean Square
JMS = Item or Rater Mean Square
EMS = Ratee x Item (Rater) Mean Square 9 9

10. Equivalence of Survey Data
Missing data rates were significantly higher for African Americans on all CAHPS items
Internal consistency reliability did not differ
Plan-level reliability estimates were significantly lower for African Americans than whites
M. Fongwa et al. (2006). Comparison of data quality for reports and ratings of ambulatory care by African American and White Medicare managed care enrollees. Journal of Aging and Health, 18, 10 10

11. Item-scale correlation matrix11 11

12. Item-scale correlation matrix12 12

13. 13

14. 14

15. Confirmatory Factor Analysis
Observed covariances comapred to covariances generated by hypothesized model
Statistical and practical tests of fit
Factor loadings
Correlations between factors 15

16. Fit Indices 1 - Non-normed fit index: Normed fit index:2
 -  2
Normed fit index:
Non-normed fit index:
Comparative fit index:
null
model  2 2 2
 
null
null -
model
df df
null
model  2
null - 1 df
null 2
 df
1 -
model
model
 - 2 df
null
null 16

17. 17

18. 18

19. 19 19

20. 20 20

21. Differential Item Functioning (2-Parameter Model)AA
White
White
Slope DIF
Location DIF AA 21
Location = uniform; Slope = non-uniform 21

22. Thank you. 22