Generalizability Theory A Brief Introduction Greg Brown UCSD

The Psychologist’s View of Reliability: The traditional True Score Theory

True Score Theory O = T + E –O is an observed test score –T is an unobservable score on an error free test –E is an unobservable error (E = O – T) Core Assumptions –T and E are independent –and errors from two different forms of a test, e 1 and e 2, are independent  2 O =  2 T +  2 E

True Score Theory (continued) Assume parallel test forms –Same test structure but different items –Means score from parallel forms are equal –Standard deviation from parallel forms are equal Correlation between the two forms is equal to the amount of variance in the observed score that can be accounted for by the true score. A regression estimate of the true score from the observed score can be derived

Cronbach’s Criticism of True Score Theory Cronbach, Rajaratnam, & Gleser (1963). Theory of generalizability: A liberalization of reliability theory. Brit J Stat Psychol. Modern factorial ANOVA designs partition sources of error as well as sources of experimenter controlled variance. The assumption in true score theory that measurement error is of a single type is illegitimate.

Measurement of Reliability: The Generalizability Theory Approach Estimate all variance components in a design, including person variance. Decide over which facets of the experimental design the experimenter wishes the person variance to generalize.

Calculating the Generalizability Coefficient G = Variance person Variance person + SE other-relevant- person-related-variance

Generalizability Theory Defines Measurement Error as a Proportion of Person Variance Standard Error = [(1-G)/G] Variance person

Components of Standard Error for Generalizability Coefficient Standard Error = [   pij  n i n j )]+ [   pi /n j ]+ [   pj /n i ]

Dependability Coefficient D = Variance person Variance person + SE all-other-effects

Two Common Stability Coefficients are Used in Generalizability Theory Generalizability Coefficient: Used to generalize to parallel experimental conditions Dependability Coefficient: Used to generalize to unequal experimental conditions

Generalizability Programs used in UCSD/VA Laboratory of Cognitive Imaging GENOVA: Generalized Analysis of Variance Developed by Brennan and colleagues at University of Iowa. Available for balanced or unbalanced univariate or multivariate designs

Expected Mean Squares defined by Variance Components EMS p =   pij  n i n j   p + n j   pi + n i   pj EMS i =   pij  n p n j   i + n j   pi + n p   ij EMS pj =   pij  n i   pj EMS j =   pij  n i n p   j + n i   pj + n p   ij EMS pi =   pij  n j   pi EMS ij =   pij  n p   ij EMS pij =   pij n i, n j, n p : sample sizes for facets i and j and the for person factor p.

GENOVA: Estimating Variance Components by Substitution Substitute calculated Mean Squares for Expected Mean Squares in previous equations. Use bottom to top substitution putting 0 wherever a negative value is found.

GENOVA: Estimating Variance Components by Direct Estimation Algorithm 1.Start with mean square of effect of interest 2.Subtract from the MS in step 1 each two way MS that includes the index for the MS in step 1 3.Add to 2 each three-way MS that includes the index for the MS in Step 1. 4.Continue to alternate subtraction and addition until all mean squares are exhausted.

GENOVA: Estimating Variance Components by Matrix Method If there are a total of M variance components for an M by M matrix of component coefficients based on produces of the relevant sample sizes. Us standard matrix algebra to estimate the variance components

END