1. C ORRECTING FOR I NDIRECT R ANGE R ESTRICTION IN M ETA -A NALYSIS : D ETERMINING THE U T D ISTRIBUTION Huy Le University of Central Florida In-Sue Oh University of Iowa

2. T HE I MPORTANCE OF C ORRECTING FOR S TUDY A RTIFACTS IN M ETA -A NALYSIS The major goal of meta-analysis is to estimate the true relationships between variables (constructs) from observed correlations. These observed correlations, however, are influenced by the effects of study artifacts  Meta-analysts need to take these effects into account in order to accurately estimate the true correlations.

3. What is Range restriction? Occurs when the variance of a variable in a sample is reduced due to pre-selection or censoring in some way (Ree, Carretta, Earles, & Albert, 1994). Effects of range restriction Statistics estimated in such a restricted sample (incumbent sample) are biased, attenuated estimates of parameters in the unrestricted population (applicant sample). R ANGE R ESTRICTION AS A S TUDY A RTIFACT

4. XY Direct Range Restriction Explicit Selection on X resulting in distortion of correlation between X and Y Indirect Range Restriction Explicit Selection on a third variable Z resulting in distortion of correlation between X and Y (which are correlated with Z); Always the case for concurrent validation studies X Z Y Multivariate Range Restriction An extension of indirect range restriction where explicit selections occur on several variables GATB Suitability scores T WO T YPES OF R ANGE R ESTRICTION

5. E FFECTS OF D IRECT AND I NDIRECT R ANGE R ESTRICTION ON C ORRELATIONS Unrestricted Correlation Rho=.60 R xx =.90 R yy =.52 Direct Range Restriction (Sr= 20%) Indirect Range Restriction (Sr= 20% on Z; R xz =.66; R yz =.30) R =.41 R =.22 R =.32

6. Direct Range Restriction: Thorndike Case II Notes: U x = 1/u X ; u X = sd x /SD x = Range restriction ratio of X (the ratio of standard deviation of the independent variable X in the restricted sample to its standard deviation in the unrestricted population). U Z = 1/u Z ; u Z = Range restriction ratio of Z (the third variable where explicit selection occurs). Indirect Range Restriction: Thorndike Case III C ORRECTION FOR R ANGE R ESTRICTION

7. Problems related to correcting for the effect of range restriction in Meta-Analysis: – Most studies are affected by indirect range restriction. – However, information required to correct for this effect of indirect range restriction (shown in the previous) is often not available. – The problem is even worst for meta-analysts who have to rely on information reported by primary researchers.

8. Recently, Hunter, Schmidt, and Le (2006) introduced a new procedure (CASE IV) to correct for range restriction. The procedure requires information about: – u T : Range restriction ration on the true score T underlying X – R xxa : Reliability of X estimated in the unrestricted population. Simulation study shows that the method is accurate (Le & Schmidt, 2006), outperforming traditional approach of using direct range restriction correction (when range restriction is actually indirect) in most situations. Using this procedure, the researchers showed that traditional estimates of the validity of the GATB were underestimated from 24% - 45%! N EW R ANGE R ESTRICTION C ORRECTION M ETHODS

9. Hunter, Schmidt, & Le (2006) model for the combined effects of indirect range restriction and measurement error: S TP XY uXuX uTuT R TX =(R XX ) 1/2 R TP R XY uSuS True validity N EW R ANGE R ESTRICTION C ORRECTION M ETHODS

10. New Method for Range Restriction (Case IV) Two key characteristics: Before applying Thorndike’s Case II, + Ut (instead of Ux) + Correction for measurement error before RR correction + Applying Thorndike’s Case II + Reintroducing unreliability in predictor to estimate true validity

11. A PPLYING THE N EW C ORRECTION A PPROACH TO M ETA -A NALYSIS Problem: The information needed to apply the new procedure is not available in every primary study. In the past, meta-analysts addressed that problem by using artifact distributions. This approach allows corrections to be made even when information of the artifacts is not available in each primary study.

12. D IFFICULTIES IN E STIMATING THE U T D ISTRIBUTION Problem when applying the artifact distribution approach to correct for indirect range restriction in meta-analysis: Need the u T artifact distribution but u T is unknown (unobservable – unlike u X )! Hunter et al. (2006) suggested u T be estimated from u X and R xx (reliability of the independent variable in the unrestricted population) using the formula:

13. THE ARTIFACT DISTRIBUTION OF U T Individual studies Study 1:R xxa1 u X1 u T1 N 1 r xy1 Study 2:R xxa2 N 2 r xy2 Study 3:u X3 N 3 r xy3 Study 4: R yy4 N 4 r xy4 …………........…….……………..…… Study k:R xxak R yyk u Xk u Tk N k r xyk Artifact Distributions ofR xx R yy u X u T Very Rare Case! Less representative! Dependent on Rxxa! Sometimes, cannot be computed even when Rxxa and Ux are simultaneously available

14. D IFFICULTIES IN E STIMATING THE U T D ISTRIBUTION Doing so, however, renders the resulting distribution of u T is highly dependent to R xxa  the assumption of independence of the artifact distributions is violated. Further, there are values in the distributions of u X and R xxa which cannot be combined. For example, when u X =.56 (equivalent to selection ratio of 40%) and R xxa =.60, we cannot estimate the corresponding value of u T. Current practice is to disregard these values.  The u T distribution estimated by the current approach may not be appropriate  Meta-analysis results may be affected.

15. E STIMATING THE U T D ISTRIBUTION Our solution: – To go backward: Instead of combining values u X and R XX in their respective distributions to estimate the values of the u T distribution, we systematically examine the appropriateness of different “plausible u T distributions” in term of how closely they can reproduce the original u X distribution when combined with the R XX distribution. – This approach is logically appropriate because u X results from u T and R XX (see Hunter et al., 2006; Le & Schmidt, 2006), not the other way around as seemingly suggested by the formula.

16. E STIMATING THE U T D ISTRIBUTION Procedure: Five steps (1) Selecting a “plausible distribution” for u T ( ). This distribution includes a number of representative values of u T, together with their respective frequencies. (2) The values of are then combined with all the values of R xx in its distribution using the following equation to calculate the corresponding u X values (equation 8, p. 422, Le & Schmidt, 2006): (3) The resulting values of u X form a distribution with frequency of each value being the product of the corresponding frequencies of and R xx in their respective distributions.

17. E STIMATING THE U T D ISTRIBUTION Procedure: (cont.) (4) This u X distribution is then compared to the observed (original) distribution of u X, based on a pre-determined criterion. If they are close enough, as determined by the criterion, the process terminates and the “plausible distribution” of specified in step (1) becomes the estimated u X distribution. Otherwise, the process continues in step (5); (5) A new plausible distribution is constructed by keeping the original values of but systematically changing their frequencies. A new iteration is then started (by returning to step 2 above). A SAS program was developed to implement the procedure ( the program is available from Huy Le).

18. E STIMATING THE U T D ISTRIBUTION Demonstration of the procedure: Note that the u X distribution for cognitive tests derived by Alexander et al. (1989) and the R xxa distribution derived by Schmidt and Hunter (1977) were used.

19. R ESULT : T HE U T D ISTRIBUTION FOR C OGNITIVE M EASURES Alexander et al. (1989)’s Distribution of u X (Cognitive) Estimated Distribution of u T uXuX Frequency Selection Ratio uTuT Frequency Selection Ratio 1.00.05100%1.00.05100%.849.1590%.807.1385.8%.766.2080%.710.1571.4%.701.2070%.632.1756.5%.649.2060%.562.1841%.603.1550%.499.2926.6%.599.0540%.432.0313.3% =.718; =.107 (Skewness =0.80; Kurtosis = 0.33) =.628; =.139 (Skewness = 0.94; Kurtosis = 0.36)

20. R ESULT : T HE U T D ISTRIBUTION FOR E DUCATIONAL T ESTS Alexander et al. (1989)’s Distribution of u X (Education) Estimated Distribution of u T uXuX Frequency Selection Ratio uTuT Frequency Selection Ratio.849.1590%.807.1185.8%.766.2080%.710.1571.4%.701.2070%.632.2356.5%.649.2060%.562.2341%.603.1550%.499.2326.6%.599.0540%.432.0513.3% =.704; =.084 (Skewness =0.28; Kurtosis = -0.84) =.607; =.105 (Skewness = 0.43; Kurtosis = 0.69)

21. D ISCUSSION Procedures to correct for indirect range restriction are necessarily complicated, but the procedure described in this paper will allow better, more accurate estimation of the u T distribution  More accurate meta-analysis results. The u T distributions estimated here can be used by researchers in their future research. Alternatively, meta-analysts can apply the current procedure to any situations where there are only sparse information about range restriction and reliabilities in their data (i.e., primary studies).

22. T HANK YOU ! Any Questions or Comments?