Reporting item response theory results Jeffrey B. Brookings Wittenberg University Presented at the SAMR/SWPA Symposium: Handy tips for communicating and reporting your findings April 5, 2013

Ph.D. Comics, 2013

Item Response Theory 1.Mathematical models that probabilistically describe the relation between a person’s response to an item and his/her standing on a latent trait. 2.The Rasch model—a “one-parameter” model (difficulty)— locates person ability and item difficulty on the same scale (logits or log odds). 3.“…a person having a greater ability than another person should have the greater probability of solving any item of the type in question, and similarly, one item being more difficult than another means that for any one person the probability of solving the second item is the greater one.” (Rasch, 1960, p. 117) 4.The purpose of Rasch analysis is to produce unidimensional measures that cover a wide range of the latent trait.

Reporting results from a Rasch analysis 1.Item and scale descriptive statistics 2.PCA of standardized residuals following extraction of the Rasch component (test for unidimensionality) 3.Item “difficulty” estimates (in logits) 4.Item fit statistics 5.Item characteristic curves (ICCs) 6.Category response curves (CRRs) 7.Person/item map 8.Person/item separation reliability

The Psychosocial Risk Factor Survey (Eichenauer, Feltz, Wilson, & Brookings, 2010) Assesses psychosocial risk factors for cardiac disease 70 items, 5-point response scale: 0 - “Strongly Agree” to 4 - “Strongly Disagree” Scales: Depression, Anxiety, Hostility, Social Isolation, and Emotional Guardedness. Analyses: Responses to the 14 Depression Scale items (340 patients from five cardiac rehabilitation programs in the Midwest)

Rasch Item Statistics

PCA of Standardized Residuals Total raw variance in observations % Raw variance explained by measures % Raw variance explained by persons % Raw variance explained by items % Raw unexplained variance (total) % Unexplained variance in 1st contrast % Unexplained variance in 2nd contrast % Unexplained variance in 3rd contrast % Unexplained variance in 4th contrast % Unexplained variance in 5th contrast %

Item characteristic curve—with 95% CI—for item 27: “My thoughts feel so scattered lately”

Item characteristic curve—with 95% CI—for item 12: “I think more about ending my life lately”

Rasch Category Responses

Person/Item Map Mean person measure = Mean item measure =.00

Reliability Person separation reliability – Analogous to Cronbach’s α; degree to which the scale differentiates persons; range 0 – 1 –For PRFS Depression:.88 Item separation reliability – Degree to which item difficulties are differentiated; range 0 – 1 –For PRFS Depression:.99

Summary of Rasch Analysis for the PRFS Depression Scale Good evidence for unidimensionality Mean point-measure r =.626 Acceptable person and item separation reliabilities (.88 and.99, respectively) Some misalignment of persons and items One mis-fitting item: #12 (“I think more about ending my life lately”)

Recommended Reading Bond, T.G., & Fox, C.M. (2007). Appling the Rasch model: Fundamental measurement in the human sciences (2 nd ed.). Mahwah, NJ: Erlbaum.