Nonequivalent Groups: Linear Methods Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and practices (2 nd ed.). New York, NY: Springer. EPSY 82251

Equating Common Population Observed Score True Score: CTT/IRT Anchor Test Observed Score True Score: CTT/IRT EPSY 82252

Nonequivalent Groups Only one test form is administered at a time. The different groups cannot be assumed to be randomly equivalent. Each group comes from a different population. To allow the linking of the two forms, a set of common items is included on each form. Common items could be internal (scored) or external (not part of the score on the form). EPSY 82253

Notation Y = score on the old form X = score on the new form V = score on the common item set EPSY 82254

The Synthetic Population The design involves two populations. But the equating function is defined for a single population. We define a synthetic population, where each of the independent populations is weighted by w 1 and w 2 where w 1 + w 2 = 1 and w 1, w 2  0 EPSY 82255

Linear Equating The linear conversion is defined by setting standardized deviation scores equal EPSY 82256

Linear Equating Function Where s indicates the synthetic population EPSY 82257

Synthetic Population Parameters  s (X) = w 1  1 (X) + w 2  2 (X)  s (Y) = w 1  1 (Y) + w 2  2 (Y) EPSY 82258

Synthetic Population Parameters In population 1, Y is not administered In population 2, X is not administered The following parameters cannot be estimated directly: EPSY 82259

Parameters Estimated from Data Form X administered to population 1 Form Y administered to population 2 EPSY

Parameters Estimated from Assumptions Form X moments in Population 2 Form Y moments in Population 1 EPSY

Nonequivalent Group Equating What sets apart these equating methods is the set of statistical assumptions used. Assumptions must be introduced to estimate the parameters not observed. Different methods employ different assumptions. EPSY

Tucker Method Assumption 1: regression of total scores on common-item scores – The regression of X on V is the same linear function for both populations 1 and 2 – The same assumption is made for the regression of Y on V The slope and regression intercept are assumed to be the same for the observed data with each population and the unobserved parameters in the other population EPSY

Tucker Method Assumption 2: conditional variances of total scores given common-item scores – Conditional variance of X given V is the same for population 1 and 2 – The same assumption is made for the conditional variance of Y given V EPSY

Tucker Method The conditional mean score on the new form increases linearly with scores on the anchor – Use a simple formula to estimate the conditional mean in the synthetic population The conditional standard error is the same at all levels of the anchor score – Estimate a single value for the conditional SE Need: Mean and SD of anchor scores in synthetic population EPSY

Tucker Method Result: the synthetic population means and variances for X and Y are adjusted to directly observable quantities. The adjustment is a function of the differences in means and variances for the common items across the two populations. If  1 (V) =  2 (V) and the synthetic parameters would equal observable means and variances. EPSY

Estimating Synthetic Pop Values  s (X) =  1 (X) - w 2 γ 1 [  1 (V) -  2 (V)]  s (Y) =  2 (Y) - w 1 γ 2 [  1 (V) -  2 (V)] Where γ 1 and γ 2 are the regression slopes from the regression of X on V for the two populations. Similarly, we can estimate EPSY

Weighting Options w 1 + w 2 = 1 and w 1, w 2  0 One could conceive of the synthetic population to be the new population: w 1 = 1 and w 2 = 0 Weights can be proportional to population sizes: w 1 = N 1 /(N 1 + N 2 ) and w 1 = N 2 /(N 1 + N 2 ) Weights can be equal, combining both populations equally: w 1 + w 2 =.5 EPSY

Tucker Issues Problems in the equating can occur if – the ability distributions between those who take the different forms differ a great deal – When the anchor is not strongly correlated with the test scores – The test scores and the anchor scores do not yield near perfect reliabilities EPSY

Levine Method Assumes that X, Y, and V are all measuring the same thing so that T X and T V as well as T Y and T V are perfectly correlated in both populations. Assumptions about true scores are made in terms of the linear regression of X on V and Y on V Assumptions about the error variances (measurement error) are made similarly EPSY

Levine The method is one that relates observed scores from form X to the scale of observed scores on form Y. However, the assumptions underlying Levine method are about the true scores, T X, T Y, and T V. These are related to observed scores as in CTT, where the error has E[ε] = 0 and ρ εT = 0. EPSY

Levine The assumptions are that X, Y, and V all measure the same thing, which implies True scores of X and V as well as True scores of Y and V are perfectly correlated in both populations 1 and 2. ρ 1 = (T X, T V ) = ρ 2 (T X, T V ) = 1 ρ 1 = (T Y, T V ) = ρ 2 (T Y, T V ) = 1 EPSY

Levine True Score Method Similar assumptions about true scores are made in this method Instead of equating observed scores, true scores are equated Although the derivations employ true scores, the equating is actually done on observed scores. EPSY

Levine True Score Method In CTT, observed score means = true score means And based on previous derivation results: Observed scores are used in place of true scores. The results do not rely on the synthetic population. EPSY

Levine True-Score Property Turns out, using observed scores in Levine’s true score equating function for the common- item nonequivalent groups design results in first-order equity, under the congeneric model. For the population with a given true score on Y, the expected value for the linearly transformed scores on X equals the expected value of the scores on Y, for all true scores on Y. EPSY

Nonequivalent Groups: Equipercentile Methods EPSY

Frequency Estimation f (x, v ) is the joint distribution; the probability that X = x and V = v. f (x) is the marginal distribution of scores on X; the probability of obtaining a score of x on X. h( v ) is the marginal distribution of scores on V. f (x| v ) is the conditional distribution of scores on Form X for examinees with a particular score on V. EPSY

The conditional expectation is It follows that EPSY

Synthetic Populations f s (x) = w 1 f 1 (x) + w 2 f 2 (x) g s (y) = w 1 g 1 (y) + w 2 g 2 (y) Because form X is not administered to population 2, f 2 (x) is not directly estimable Because form Y is not administered to population 1, g 1 (y) is not directly estimable EPSY

Assumptions For both Form X and Form Y, the conditional distribution of total score given each score, V = v, is the same in both populations. EPSY

Estimation These assumptions can be used to find expressions for f 2 (x) and g 1 (y) using quantities for which direct estimates are available. EPSY

Chained Equipercentile Equating Angoff (1971) Form X scores are converted to scores on the common items using examinees from population 1 Scores on the common items are equated to form Y scores using examinees from Population 2. These conversions are chained together to produce a conversion of Form X to Form Y EPSY