Presentation is loading. Please wait.

Presentation is loading. Please wait.

Psychometric testing and validation (Multi-trait scaling and IRT)

Similar presentations


Presentation on theme: "Psychometric testing and validation (Multi-trait scaling and IRT)"— Presentation transcript:

1 Psychometric testing and validation (Multi-trait scaling and IRT)
Ron D. Hays, Ph.D. November 7, 2018 (1:00-2:30 pm) Instrument Design Workshop (7th Floor Conf. Room)

2 Acknowledgements Yelba Castellon, Jonathan Grotts, Kenrick Duru
Prediabetes Informed Decisions and Education (PRIDE) study included a measure of patient activation. My time supported by NIA Resource Centers for Minority Aging Research (P30-AG021684).

3 Patient Activation Measures
“Some authors charge fees for using published instruments; fees can include a licensing fee, administration fee, and/or fee for obtaining scoring algorithms. For example, use of the Patient Activation Measure (PAM), developed with funding by the Robert Wood Johnson Foundation, requires a licensing fee. As a result, researchers are looking for other alternatives such as the Altarium Consumer Engagement (ACE) Measure” (p. 108). Hays, R. D., Weech-Maldonado, R., Teresi, J. A., Wallace, S. P., & Stewart, A. L. (2018). Copyright restrictions versus open access to survey instruments. Medical Care, 56 (2),

4 Altarum Consumer Engagement (ACE) Questionnaire
Gruman et al (2010) “actions individuals must take to obtain the greatest benefit from the health care services available to them” 21 items/4 multi-item scales Commitment (6 items) Informed choice (5 items) Navigation (5 items) Ownership (5 items) Likert response scale Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly agree --> Duke, C. C. et al. (2015). Patient, 8,

5 12 ACE items q1) I spend a lot of time learning about health.
q2) Even when life is stressful, I know I can continue to do the things that keep me healthy. q3) I feel comfortable talking to my doctor about my health. q4) When I work to improve my health, I succeed. q5) I have brought my own information about my health to show my doctor. q6) When choosing a new doctor, I look for information online. q7) I can stick with plans to exercise and eat a healthy diet. q8) I compare doctors using official ratings about how well their patients are doing. q9) I have lots of experience using the health care system. q10) When choosing a new doctor, I look for official ratings based on patient health. q11) Different doctors give different advice; its up to me to choose whats right for me. q12) I handle my health well.

6 Aims Evaluate the dimensionality of 12 of the ACE items in a sample of 515 pre-diabetics. 58 years old (median), 56% female, 46% white, 15% AA, 19% Asian/Pacific Islander, 20% other Estimate ACE scores using item response theory (IRT) graded response model. Examine item category response curves, item and scale information. Assess construct validity of ACE scale scores.

7 Item-Level Statistics

8 Reliability Formulas 0.74 Model Reliability Intraclass Correlation
Two-way random Two-way mixed One-way BMS = Between Ratee Mean Square N = n of ratees WMS = Within Mean Square k = n of items or raters JMS = Item or Rater Mean Square EMS = Ratee x Item (Rater) Mean Square 8

9 Exploratory Factor Analysis
Polychoric correlations Correlation between two underlying normally distributed continuous variables. Three factors suggested by scree plot and Tucker and Lewis reliability coefficient = 0.95

10 Exploratory Factor Loadings and Correlations (Promax Rotation)
Commitment to Health Doctor Choice Learning about Health

11 Factor 1: Commitment to Health
q2) Even when life is stressful, I know I can continue to do the things that keep me healthy. q4) When I work to improve my health, I succeed. q7) I can stick with plans to exercise and eat a healthy diet. q12) I handle my health well. All 4 of the above items are in 6-item ACE commitment scale

12 Factor 2: Doctor Choice q6) When choosing a new doctor, I look for information online. q8) I compare doctors using official rating about how well their patients are doing. q10) When choosing a new doctor, I look for official ratings based on patient health. All 3 of the above items are in 5-item ACE informed choice scale

13 Factor 3: Learning About Health
q1) I spend a lot of time learning about health. q3) I feel comfortable talking to my doctor about my health. q5) I have brought my own information about my health to show my doctor. q9) I have lots of experience using the health care system. q11) Different doctors give different advice; its up to me to choose what's right for me. Four of the above items are in the 5-item ACE navigation scale. Q1 is in the the 5-item ACE informed choice scale.

14 Multi-trait Scaling (n = 483, SE = 0.05)

15 CFA Fit Indices 1 - Non-normed fit index: Normed fit index:
2  -  2 Normed fit index: Non-normed fit index: Comparative fit index: null model 2 2 2   null null - model df df null model 2 null - 1 df null 2  df 1 - model model  - 2 df null null RMSEA = SQRT (λ2 – df)/SQRT (df (N – 1)) CFI >=0.95 and RMSEA <=0.06 15

16 Confirmatory Factor Analysis (Polychoric* Correlations)
CFI = 0.961, RMSEA = 0.062 Commitment to Health Doctor Choice Learning about Health

17 Bifactor Model Loadings (CFI = 0.91 and RMSEA = 0.08)
Item General Group 1 Doctor Choice Group 3 Q2 0.65 0.33 Q7 0.24 Q6 0.28 0.50 Q8 0.43 0.70 Q10 0.34 0.77 Q1 0.51 0.26 Q3 0.41 Q9 0.30 0.66 Q11 Q4 0.62 Q5 0.40 Q12 0.64

18 People and Items on Same z-score metric
Person 1 Person 2 Person 3 -3 3

19 Item Response Theory (IRT)
IRT graded response model estimates relationship between a person’s response Yi to the question (i) and his or her level on the latent construct (): e a(-b)/(1 + e a(-b)) bik = how difficult it is to have a score of k or more on item (i). ai = item discrimination.

20 Item Characteristic Curve

21 Commitment to Health 0 =Strongly disagree; 1 = disagree; 2 = neither agree nor disagree; 3 = agree; 4 = strongly agree

22 Doctor Choice 0 =Strongly disagree; 1 = disagree; 2 = neither agree nor disagree; 3 = agree; 4 = strongly agree

23 Learning about Health Here we see Q3 (I feel comfortable talking to my doctor about my health) is problematic. “Agree” (brown) was picked by a lot of people with high activation and almost no one with high levels of activation picked strongly agree (purple). Also a large proportion of people picked “neither agree nor disagree” so this question may not be performing the way we want, suggesting it may be a good idea to remove it and rescore without it. 0 =Strongly disagree; 1 = disagree; 2 = neither agree nor disagree; 3 = agree; 4 = strongly agree

24 Commitment to Health (Slopes)
q2) Even when life is stressful, I know I can continue to do the things that keep me healthy. (1.72) q4) When I work to improve my health, I succeed. (1.47) q7) I can stick with plans to exercise and eat a healthy diet. (2.05) q12) I handle my health well. (1.37)

25 Doctor Choice (Slopes)
q6) When choosing a new doctor, I look for information online. (1.29) q8) I compare doctors using official ratings about how well their patients are doing. (2.69) q10) When choosing a new doctor, I look for official ratings based on patient health. (3.28)

26 Learning About Health (Slopes)
q1) I spend a lot of time learning about health. (1.08) q3) I feel comfortable talking to my doctor about my health. (0.93) q9) I have lots of experience using the health care system. (1.48) q11) Different doctors give different advice; its up to me to choose what's right for me. (0.97)

27 Ordinal Alpha (Diagonal) and Product-moment Correlations Among ACE Scales (Graded Response Model)
Commitment to Health Doctor Choice Learning about Health 0.75 0.32 0.78 0.39 0.28 0.59 .

28 Internal Consistency Reliability (Coefficient Alpha)
(MSbms – MSems)/MSbms Ordinal alpha

29 Reliability = (Info – 1) / Info

30 Commitment to Health For example, in this curve, maximum reliability is around 3 so roughly R= (3-1/3=2/3 or .67)

31 Doctor Choice For doctor choice we can use 4-1/4=3/4=0.75

32 Learning About Health 1.25-1/1.25=0.2

33 Excellent or Very Good Health
Statistically Significant Spearman Correlations of ACE Scale Scores with Other Variables D > High School PHQ-5 >4 Excellent or Very Good Health Obese (BMI>= 30 kg/m2) Commitment to Health (ACE) 0.12 -0.31 0.31 -0.18 Commitment to Health (ACE*) 0.14 -0.27 0.20 Doctor Choice (ACE) 0.08 Doctor Choice (ACE*) 0.09 Learning about Health (ACE) Learning about Health (ACE*) Cohen’s effect size rules of thumb (d = 0.2, 0.5, and 0.8): small = 0.100; medium = 0.243, and large = r = d / [(d2 + 4).5] = 0.8 / [( ).5] = 0.8 / [( ).5] = 0.8 / [( 4.64).5] = / = 0.371

34 https://altarum.org/publications/right-place-right-time


Download ppt "Psychometric testing and validation (Multi-trait scaling and IRT)"

Similar presentations


Ads by Google