ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132) Washington, DC 1

The ASVAB is the principal instrument for selection and classification in the U.S. Armed Forces. Assumption: measurement invariance across full range of scores. Relevance:what is the “population of interest” of the ASVAB? Recent studies cast doubt on this assumption. 2

Two related phenomena suggest a changing factor structure along the dimension of general intelligence (g): The g factor gets smaller in high-g samples Cognition tests have smaller loadings on “g” in high-g samples 3

Spearman (1927) noticed already a decrease in the positive manifold of cognition variables at higher g levels. differentiation of intelligence Spearman’s explanation: 4

5 The present study adheres to an alternative explanation: The phenomenon follows from the Pearson-Lawley selection rules. an underlying selection process changes the variance-covariance structure and the mean structure

6 One phenomenon: Decrease in positive manifold of cognition variables in high-g samples Two explanations: differentiation of intelligence selection effects

Consequences of: differentiation: selection effects: structure is changing underlying structure invariant 7

8 Critical developments in psychometric theory: Meredith (1964) showed that both the covariance structure and mean structure change if samples are selected based on one or more latent variables (e.g., the g factor). Meredith (1965) developed procedures to derive the single best fitting (i.e., invariant) factor pattern derived from sets of factors obtained on populations differing on a latent variable. Jöreskog (1971) formalized this viewpoint as an extension of the common factor model for a parent population to multiple groups based on one or more latent variables in the model.

(df) Measurement Invariance: If we compare groups, or individuals of different groups, then the expected value of test scores of a person of a given level of ability should be independent of membership of these groups (Mellenbergh, 1989). In formule: f (Y | η, ν) = f (Y | η) y 1ij = τ 1i + λ 1i η ij + ε 1ij f depends on the measurement model of choice: 9

10 y 1ij = τ 1i + λ 1i η ij + ε 1ij change invariant

Unequal intercepts 11

ratings cluster1 2 η 12 η 11 Unequal factor loadings 12

13 parent population (N = 48,222) a-select sample (n=1,000) hi-g sample (n=600) av-g sample (n=600) lo-g sample (n=600) Statistical Experiment:

14 parent population (N = 48,222) a-select sample ( n=1,000) hi-g sample (n=600) av-g sample (n=600) lo-g sample (n=600) Statistical Experiment: determine factor structure and then sample

Eigenvalues from a-select samples drawn from the parent population of Air Force recruits 15

ASVAB tests and their measurement claims 16

Model 1: A hierarchical model of “g” 17

Model 2: A “g as first principal factor” model 18

19 parent population (N = 48,222) a-select sample (n=1,000) hi-g sample (n=600) av-g sample (n=600) lo-g sample (n=600) Statistical Experiment: determine factor structure and sample

20 sampleNms.d.skewnesskurtosis random g-hi g-av g-lo Distributional properties of samples generated from the parent population based on a latent variable "g"

ASVAB tests mean scores in samples with different levels of "g" 21

22 Average correlation lower with lower variance

23 The effects of selection based on the latent variable “g” on the variance of ASVAB tests

Sequence of MCFA model fits and goodness of fit indices 24

Non-standardized MCFA factor loadings. 25

Standardized MCFA factor loadings 26

27 y 1ij = τ 1i + λ 1i η ij + ε 1ij change should remain invariant

28 y 1ij = τ 1i + λ 1i η ij + ε 1ij change not invariant

29 Discussion: ASVAB is measurement invariant in a limited sense: only factor loadings are invariant across different levels of “g”. (weak factorial invariance). ASVAB seems to be measuring too many factors with too few tests. more factors than eigenvalues larger than 1. many tests have communalities < intercepts could not be constrained to be equal (indicating: other factors influence test scores).