1. Introduction to Psychometric Analysis of Survey Data
Ron D. Hays, Ph.D.
July 10, 2003 (10:30-12:30pm)

2. Lori Shiotani
What kind of data collection errors are possible?

3. Data Collection Errors
Do respondents represent underlying population?
Coverage Error (every person in population is not included in the sampling frame)
Sampling Error (only some members of the population are sampled)
Non-response error (those who response are different from those who do)
Are inaccurate answers given to survey questions?
Measurement error

4. What’s a Good Measure? It is practical to use (feasibility)
Same person gets same score (reliability)
Different people get different scores (validity)
People get scores you expect (validity)

5. Peter Chin
How are good measures developed?

6. How Are Good Measures Developed?
Review literature
Expert input (patients and clinicians)
Define constructs you are interested in
Draft items (item generation)
Pretest
Cognitive interviews
Field and pilot testing
Revise and test again
Translate/harmonize across languages

7. Scales of Measurement and Their Properties
Property of Numbers
Type of
Scale
Equal
Interval
Rank Order
Absolute 0
Nominal No No No
Ordinal Yes No No
Interval Yes Yes No
Ratio Yes Yes Yes
0 1 2 3 4 5

8. Measurement Range for Health Services Measures
Nominal
Ordinal
Interval
Ratio

9. Indicators of Acceptability
Response rate
Missing data (item, scale)
Administration time

10. Variability • All scale levels are represented
• Distribution approximates bell-shaped "normal"

11. Measurement Error observed = true score systematic error random error+ +
(bias)

12. Flavors of Reliability
•Test-retest (administrations)
Intra-rater (raters)
Internal consistency (items)

13. Test-retest Reliability of MMPI 317-362
True
False
169 15
184
True
MMPI 362
False 21 95
116
190
110
I am more sensitive than most other people. (r = 0.75)

14. Kappa Coefficient of Agreement (Corrects for Chance)
(observed - chance)
kappa =
(1 - chance)

15. Example of Computing KAPPA
Rater A
Row
Sum 1 2 3 4 5 1 2 3 4 5 1 1 2 2
Rater B 2 2 2 3 2 10
Column Sum

16. Example of Computing KAPPA (Continued)
(1 x 2) + (3 x 2) + (2 x 2) + (2 x 2) + (2 x 2) P c = =
0.20
(10 x 10) 9 P
obs. = =
0.90 10
Kappa =
0.87 =

17. Guidelines for Interpreting Kappa
Conclusion Kappa Conclusion Kappa
Poor < Poor < 0.0
Fair Slight
Good Fair
Excellent > Moderate
Substantial
Almost perfect
Fleiss (1981) Landis and Koch (1977)

18. Ratings of Height of Houseplants
Baseline
Height
Follow-up
Height
Experimental
Condition
Plant A1 R R A2 R R B1 R R B2 R R C1 R R

19. Ratings of Height of Houseplants (Cont.)
Baseline
Height
Follow-up
Height
Experimental
Condition
Plant C2 R R D1 R R D2 R R E1 R R E2 R R

20. Reliability of Baseline Houseplant Ratings
Ratings of Height of Plants: 10 plants, 2 raters
Baseline Results
Source DF SS MS F
Plants
Within
Raters
Raters x Plants
Total

21. Sources of Variance in Baseline Houseplant Height
Source dfs MS
Plants (N) (BMS)
Within (WMS)
Raters (K) (JMS)
Raters x Plants (EMS)
Total 19

22. Intraclass Correlation and Reliability
Model Reliability Intraclass Correlation
One-Way MS MS MS MS
MS MS (K-1)MS
Two-Way MS MS MS MS
Fixed MS MS (K-1)MS
Two-Way N (MS MS ) MS MS
Random NMS MS MS MS (K-1)MS K(MS MS )/N
BMS
WMS
BMS
WMS
BMS
BMS
WMS
BMS
EMS
BMS
EMS
BMS
EMS
EMS
BMS
EMS
BMS
EMS
BMS
JMS
EMS
BMS
EMS
JMS
EMS

23. Summary of Reliability of Plant Ratings
Baseline Follow-up
RTT RII RTT RII
One-Way Anova
Two-Way Random Effects
Two-Way Fixed Effects
Source Label Baseline MS
Plants BMS
Within WMS
Raters JMS
Raters X Plants EMS

24. Cronbach’s Alpha Respondents (BMS) 4 11.6 2.9 Items (JMS) 1 0.1 0.1
Resp. x Items (EMS)
Total
Source SS MS df
Alpha =
= = 0.62
2.9
2.9

25. Alpha by Number of Items and Inter-item Correlations_ K r
alphast = _ 1 + r
(K - 1 ) K =
number of items in scale

26. Alpha for Different Numbers of Items and Homogeneity
Average Inter-item Correlation ( r )
Number
of Items (K) .0 .2 .4 .6 .8
1.0

27. Spearman-Brown Prophecy Formula( )
N • alpha x
alpha = y
1 + (N - 1) * alpha x
N = how much longer scale y is than scale x

28. Reliability Minimum Standards
0.70 or above (for group comparisons)
0.90 or higher (for individual assessment)
SEM = SD (1- reliability)1/2

29. Reliability of a Composite Score

30. Hypothetical Multitrait/Multi-Item Correlation Matrix

31. Multitrait/Multi-Item Correlation Matrix for Patient Satisfaction Ratings
Technical Interpersonal Communication Financial
Technical * 0.63† 0.67† * 0.54† 0.50† * † * † * 0.60† 0.56† * 0.58† 0.57† 0.23
Interpersonal * 0.63† † 0.58* 0.61† † 0.65* 0.67† † 0.57* 0.60† * 0.58† † 0.48* 0.46† 0.24
Note – Standard error of correlation is Technical = satisfaction with technical quality. Interpersonal = satisfaction with the interpersonal aspects. Communication = satisfaction with communication. Financial = satisfaction with financial arrangements. *Item-scale correlations for hypothesized scales (corrected for item overlap). †Correlation within two standard errors of the correlation of the item with its hypothesized scale.

32. Forms of Validity Content Criterion Construct Validity
Measure’s relationships with other things are consistent with hypotheses/theory.
Includes responsiveness to change

33. Relative Validity Example
Severity of Heart Disease
Relative
None
Mild
Severe
F-ratio
Validity
Scale #1
Scale #2
Scale #3

34. Responsiveness to Change
Measures should reflect true change
• Evaluating responsiveness requires an external
indicator of change (anchor)

35. Responsiveness Indices
(1) Effect size (ES) = D/SD
(2) Standardized Response Mean (SRM) = D/SD†
(3) Guyatt responsiveness statistic (RS) = D/SD‡
D = raw score change in “changed” group;
SD = baseline SD;
SD† = SD of D;
SD‡ = SD of D among “unchanged”

36. Kinds of Anchors Self-report Clinician or other report
Clinical parameter
Clinical intervention

37. Self-Report Anchor
Overall has there been any change in your asthma since the beginning of the study?
Much improved; Moderately improved; Minimally improved
No change
Much worse; Moderately worse; Minimally worse

38. Examples of Other Anchors
Clinician report
How is Jan’s physical health now compared to 4 weeks ago?
Clinical parameter
Change from CDC Stage A to B
Became seizure free
Clinical intervention
Before and after Prozac

39. Change and Responsiveness in PCS Depends on Treatment

40. Change and Responsiveness in MCS Depends on Treatment

41. Magnitude of HRQOL Change Should Parallel Underlying Change

42. Minimally Important Difference (MID)
Some differences between groups or over time may be so small in magnitude that they are not important.
Smallest difference in score that is worth caring about (important).
Change large enough for a clinician to base treatment decisions upon it.

43. Identifying the MID People who report a “minimal” change
How is your physical health now compared to 4 weeks ago?
Much improved; Moderately Improved;
Minimally Improved;
No Change;
Minimally Worse;
Moderately Worse; Much Worse

44. MID Varies by Anchor
693 RA clinical trial participants evaluated at baseline and 6-weeks post-treatment.
Five anchors: 1) patient global self-report; 2) physician global report; 3) pain self-report; 4) joint swelling; 5) joint tenderness
Kosinski, M. et al. (2000). Determining minimally important changes in generic and disease-specific health-related quality of life questionnaires in clinical trials of rheumatoid arthritis. Arthritis and Rheumatism, 43,

45. Changes in SF-36 Scores Associated with Minimal Change in Anchors
Scale
Self-R
Clin.-R
Pain
Swell
Tender
Mean PF 8 6
Role-P 21 20 11 13 16 15 12 7 GH 4 2 3 1
EWB 5
Role-E 18
13* SF 9 10 EF
PCS
3.5*
MCS

46. Changes in SF-36 Scores Associated with Minimal Change in Anchors
Scale
Mean (ES)
Range SD
Range/SD PF
8 (.4)
2 ( 6 – 8) 20
.10
Role-P
16 (.4)
10 (11-21) 40
.25
Pain
11 (.5)
8 ( 7-15)
.40 GH
2 (.1)
3 ( 1- 4)
.15
EWB
4 (.2)
6 ( 1- 7)
.30
Role-E
13 (.2)
10 ( 8-18) SF
9 (.5)
4 ( 8-12)
.20 EF
8 (.4)
6 ( 5-11)
PCS
3 (.3)
1 ( 3- 4) 10
MCS
3 ( 2- 5)

47. This class was supported in part by the UCLA/DREW Project EXPORT, National Institutes of Health, National Center on Minority Health & Health Disparities, (P20-MD ) and the UCLA Center for Health Improvement in Minority Elders / Resource Centers for Minority Aging Research, National Institutes of Health, National Institute of Aging, (AG ).