Psychometric testing and validation (Multi-trait scaling and IRT) Ron D. Hays, Ph.D. November 7, 2018 (1:00-2:30 pm) Instrument Design Workshop (7th Floor Conf. Room) https://drhays.dgsom.ucla.edu/pages/presentations

Acknowledgements Yelba Castellon, Jonathan Grotts, Kenrick Duru Prediabetes Informed Decisions and Education (PRIDE) study included a measure of patient activation. My time supported by NIA Resource Centers for Minority Aging Research (P30-AG021684).

Patient Activation Measures https://www.insigniahealth.com/products/pam-survey “Some authors charge fees for using published instruments; fees can include a licensing fee, administration fee, and/or fee for obtaining scoring algorithms. For example, use of the Patient Activation Measure (PAM), developed with funding by the Robert Wood Johnson Foundation, requires a licensing fee. As a result, researchers are looking for other alternatives such as the Altarium Consumer Engagement (ACE) Measure” (p. 108). Hays, R. D., Weech-Maldonado, R., Teresi, J. A., Wallace, S. P., & Stewart, A. L. (2018). Copyright restrictions versus open access to survey instruments. Medical Care, 56 (2), 107-110.

Altarum Consumer Engagement (ACE) Questionnaire Gruman et al (2010) “actions individuals must take to obtain the greatest benefit from the health care services available to them” 21 items/4 multi-item scales Commitment (6 items) Informed choice (5 items) Navigation (5 items) Ownership (5 items) Likert response scale Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly agree --> Duke, C. C. et al. (2015). Patient, 8, 559-568.

12 ACE items q1) I spend a lot of time learning about health. q2) Even when life is stressful, I know I can continue to do the things that keep me healthy. q3) I feel comfortable talking to my doctor about my health. q4) When I work to improve my health, I succeed. q5) I have brought my own information about my health to show my doctor. q6) When choosing a new doctor, I look for information online. q7) I can stick with plans to exercise and eat a healthy diet. q8) I compare doctors using official ratings about how well their patients are doing. q9) I have lots of experience using the health care system. q10) When choosing a new doctor, I look for official ratings based on patient health. q11) Different doctors give different advice; its up to me to choose whats right for me. q12) I handle my health well.

Aims Evaluate the dimensionality of 12 of the ACE items in a sample of 515 pre-diabetics. 58 years old (median), 56% female, 46% white, 15% AA, 19% Asian/Pacific Islander, 20% other Estimate ACE scores using item response theory (IRT) graded response model. Examine item category response curves, item and scale information. Assess construct validity of ACE scale scores.

Item-Level Statistics

Reliability Formulas 0.74 Model Reliability Intraclass Correlation Two-way random Two-way mixed One-way BMS = Between Ratee Mean Square N = n of ratees WMS = Within Mean Square k = n of items or raters JMS = Item or Rater Mean Square EMS = Ratee x Item (Rater) Mean Square 8

Exploratory Factor Analysis Polychoric correlations Correlation between two underlying normally distributed continuous variables. Three factors suggested by scree plot and Tucker and Lewis reliability coefficient = 0.95

Exploratory Factor Loadings and Correlations (Promax Rotation) Commitment to Health Doctor Choice Learning about Health

Factor 1: Commitment to Health q2) Even when life is stressful, I know I can continue to do the things that keep me healthy. q4) When I work to improve my health, I succeed. q7) I can stick with plans to exercise and eat a healthy diet. q12) I handle my health well. All 4 of the above items are in 6-item ACE commitment scale

Factor 2: Doctor Choice q6) When choosing a new doctor, I look for information online. q8) I compare doctors using official rating about how well their patients are doing. q10) When choosing a new doctor, I look for official ratings based on patient health. All 3 of the above items are in 5-item ACE informed choice scale

Factor 3: Learning About Health q1) I spend a lot of time learning about health. q3) I feel comfortable talking to my doctor about my health. q5) I have brought my own information about my health to show my doctor. q9) I have lots of experience using the health care system. q11) Different doctors give different advice; its up to me to choose what's right for me. Four of the above items are in the 5-item ACE navigation scale. Q1 is in the the 5-item ACE informed choice scale.

Multi-trait Scaling (n = 483, SE = 0.05)

CFA Fit Indices 1 - Non-normed fit index: Normed fit index: 2  -  2 Normed fit index: Non-normed fit index: Comparative fit index: null model  2 2 2   null null - model df df null model  2 null - 1 df null 2  - df 1 - model model  - 2 df null null RMSEA = SQRT (λ2 – df)/SQRT (df (N – 1)) CFI >=0.95 and RMSEA <=0.06 15

Confirmatory Factor Analysis (Polychoric* Correlations) CFI = 0.961, RMSEA = 0.062 Commitment to Health Doctor Choice Learning about Health

Bifactor Model Loadings (CFI = 0.91 and RMSEA = 0.08) Item General Group 1 Doctor Choice Group 3 Q2 0.65 0.33   Q7 0.24 Q6 0.28 0.50 Q8 0.43 0.70 Q10 0.34 0.77 Q1 0.51 0.26 Q3 0.41 Q9 0.30 0.66 Q11 Q4 0.62 Q5 0.40 Q12 0.64

People and Items on Same z-score metric Person 1 Person 2 Person 3 -3 3

Item Response Theory (IRT) IRT graded response model estimates relationship between a person’s response Yi to the question (i) and his or her level on the latent construct (): e a(-b)/(1 + e a(-b)) bik = how difficult it is to have a score of k or more . on item (i). ai = item discrimination.

Item Characteristic Curve

Commitment to Health 0 =Strongly disagree; 1 = disagree; 2 = neither agree nor disagree; 3 = agree; 4 = strongly agree

Doctor Choice 0 =Strongly disagree; 1 = disagree; 2 = neither agree nor disagree; 3 = agree; 4 = strongly agree

Learning about Health Here we see Q3 (I feel comfortable talking to my doctor about my health) is problematic. “Agree” (brown) was picked by a lot of people with high activation and almost no one with high levels of activation picked strongly agree (purple). Also a large proportion of people picked “neither agree nor disagree” so this question may not be performing the way we want, suggesting it may be a good idea to remove it and rescore without it. 0 =Strongly disagree; 1 = disagree; 2 = neither agree nor disagree; 3 = agree; 4 = strongly agree

Commitment to Health (Slopes) q2) Even when life is stressful, I know I can continue to do the things that keep me healthy. (1.72) q4) When I work to improve my health, I succeed. (1.47) q7) I can stick with plans to exercise and eat a healthy diet. (2.05) q12) I handle my health well. (1.37)

Doctor Choice (Slopes) q6) When choosing a new doctor, I look for information online. (1.29) q8) I compare doctors using official ratings about how well their patients are doing. (2.69) q10) When choosing a new doctor, I look for official ratings based on patient health. (3.28)

Learning About Health (Slopes) q1) I spend a lot of time learning about health. (1.08) q3) I feel comfortable talking to my doctor about my health. (0.93) q9) I have lots of experience using the health care system. (1.48) q11) Different doctors give different advice; its up to me to choose what's right for me. (0.97)

Ordinal Alpha (Diagonal) and Product-moment Correlations Among ACE Scales (Graded Response Model) Commitment to Health Doctor Choice Learning about Health 0.75 0.32 0.78 0.39 0.28 0.59 .

Internal Consistency Reliability (Coefficient Alpha) (MSbms – MSems)/MSbms Ordinal alpha http://support.sas.com/resources/papers/proceedings14/2042-2014.pdf https://drhays.dgsom.ucla.edu/pages/programs_utilities

Reliability = (Info – 1) / Info

Commitment to Health For example, in this curve, maximum reliability is around 3 so roughly R= (3-1/3=2/3 or .67)

Doctor Choice For doctor choice we can use 4-1/4=3/4=0.75

Learning About Health 1.25-1/1.25=0.2

Excellent or Very Good Health Statistically Significant Spearman Correlations of ACE Scale Scores with Other Variables D > High School PHQ-5 >4 Excellent or Very Good Health Obese (BMI>= 30 kg/m2) Commitment to Health (ACE) 0.12 -0.31 0.31 -0.18 Commitment to Health (ACE*) 0.14 -0.27 0.20 Doctor Choice (ACE) 0.08 Doctor Choice (ACE*) 0.09 Learning about Health (ACE) Learning about Health (ACE*) Cohen’s effect size rules of thumb (d = 0.2, 0.5, and 0.8): small = 0.100; medium = 0.243, and large = 0.371 r = d / [(d2 + 4).5] = 0.8 / [(0.82 + 4).5] = 0.8 / [(0.64 + 4).5] = 0.8 / [( 4.64).5] = 0.8 / 2.154 = 0.371

https://altarum.org/publications/right-place-right-time drhays@ucla.edu