Unidimensionality (U): What’s it good for in applied research? Richard N. Jones, Sc.D. Hebrew Rehabilitation Center for Aged Research and Training Institute Boston, MA jones@mail.hrca.harvard.edu Paul K. Crane M.D., M.P.H. University of Washington Seattle, WA pcrane@u.washington.edu Psychometrics Workshop September 20-25, 2004

Overview of this talk Introduction to concept of Unidimensionality (U) Appeal to authority What pioneers and prominent experts and authors and others have to say about U Addressing the problem of U empirically Discussion

Issues Around Unidimensionality What does U mean? Why is U important? What is the test for? What is the analysis for? DIF studies Item banking Can U be determined empirically? What to if you don’t think you have a U test? How to respond to a reviewer with concerns about U?

Other Sources Gould, S. (1981). The Mismeasure of man. New York: WW Norton & Company. Lord, F. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers. Lord, F., & Novick, M. (1968). Latent traits and item characteristic functions (Chapter 16). In Statistical Theories of Mental Test Scores (pp. 358-393). Reading, MA: Addison-Wesley. Stout, W. (2002). Psychometrics: From practice to theory and back: 15 years of nonparametric multidimensional IRT, DIF/test equity, and skills diagnostic assessment. Psychometrika, 67(4), 485-518. McDonald, R. (1999). Test Theory: A Unified Treatment. Mahwah, NJ: Erlbaum. Thissen, D. (2004) Comments on Developing Tailored Instruments: Item Banking and Computer Adaptive Assessment. Conference Proceedings (Advances in Health Outcomes Measurement). National Institutes of Health. Bethesda, Maryland. http://outcomes.cancer.gov/conference/irt/thissen.pdf

Lord and Novick (’68) on U Actual existence of traits not necessary for IRT ...nowhere is there any implication that traits exist in any physical or physiological sense. It is sufficient that a person behave as if he [or she] were in possession of a certain amount of each of a number of relevant traits and that he [or she] behave as if these amounts determined his [or her] behavior. (p 358)

Embertson and Reise (‘00) on U U IRT models = a single latent trait is sufficient to characterize individual differences, for example Single common factor Multiple factors proportionally loading in items Not appropriate when Multiple factors load differentially in different items Persons (groups) differ in factors that load in items (e.g., test-taking strategies, interpretations of items, substantive factors that load in items) Mathematics ability and reading ability in written math items Working memory and concentration in certain cognitive tests

Stout’s Essential U U is, essentially, local independence Strong local independence Probability of responding u is independent of other test item responses, conditional on q Weak local independence The conditional covariance (controlling for some trait q) for all test items is 0 Essential Unidimensionality The sum of the conditional covariance (controlling for some latent trait q) of all pairs of test items from a test approaches 0 as the test length approaches infinity Stout, W. (2002). Psychometrics: From practice to theory and back: 15 years of nonparametric multidimensional IRT, DIF/test equity, and skills diagnostic assessment. Psychometrika, 67(4), 485-518.

Jones on U Factor Analysis solutions are sample dependent Think about your data Use parcels/testlets/composites where appropriate and possible Understand logical dependencies among test items Look at model residuals Use an empirical device to support an argument for U Not one is universally accepted Argument for U is sample dependent Sample of persons Sample of behaviors Model misspecification (Data-driven versus a priori latent trait models can lead to different conclusions)

McDonald’s Device for U ‘Hierarchical’ Factor analysis Aka General-specific factor model Bi-factor model Others? The hierarchical model provides useful evidence of U

McDonald argues that evidence for unidimensionality is supported by observing that the loadings of the items in the general factor are larger than the loadings of the in the specific factors.