PSY 1950 Outliers, Missing Data, and Transformations September 22, 2008

On Suspecting Fishiness Looking for outliers, gaps, and dips –e.g., tests of clairvoyance When gaps or dips are hypothesized –e.g., is dyslexia a distinct entity Cliffs –e.g., differences between rating of ingroup and outgroup Peaks –e.g., the blackout and baby boom The occurrence of impossible scores

Visualize your data! “make friends with your data” –Rosenthal “don’t becomes lovers with your data” –Me Statistics condense data View raw data graphically –Frequency distribution graphs –Scatter plots

Outliers Extreme scores Come from samples other than those of interest Can lead to Type I and II errors

Outlier Detection Graph –Box plots –Scatter plots Numerical criterion –Extremity (central tendency +/- spread) Outside fences –lower: Q1 - 3(Q3 - Q1) –upper: Q3 + 3(Q3 - Q1) z-score –Probability (Extremity + # measurements) Chauvenat’s/Peirce’s criterion, Grubb’s test –Absolute cutoff

Outlier Analysis Determine nature of impact –Quantitative Changes numbers, not inferences –Qualitative Changes numbers and inferences Consider source of outlier –Quantitative Same underlying mechanism/sample –Qualitative Different underlying mechanisms/samples –e.g., digit span = 107, simple RT = 1200 ms

Outlier Coping Options –Retain –Remove –Reduce Windsorize Normalizing transformation Considerations –Impact/Source –Convention –Believability Justification Replication

Transformations Linear “rescaling” –unit conversion e.g., # items correct, # items wrong e.g., standardization Curvilinear “reexpression” –variable conversion e.g., time (sec/trial) to speed (trials/sec) e.g., normalization

Standardization Why standardize data? –Intra-distribution statistics You got 8 questions wrong on one exam You were one standard deviation below the mean –Inter-distribution statistics You got 8 questions wrong on the midterm and 5 questions wrong on the final Aggregation: Overall, you were one standard deviation below the mean Comparison: You did better on the midterm than the final

z-score # standard deviations above/below the mean

Normal Distributions “…normality is a myth; there never was, and never will be, a normal distribution.” –Geary (1947) “Experimentalists think that it is a mathematical theorem while the mathematicians believe it to be an experimental fact.” –Lippman (1917)

Normalization Why normalize DV? –Meet statistical assumption of normality in situations when it matters Small n Unequal n One-sample t and z tests –Increase power Why NOT normalize DV? –Interpretability –Affects measurement scale

Tests of Normality Frequency distribution Skew/kurtosis statistics Kolmorogov-Smirnov test Probability plots (e.g., P-P plot)

Types of Curvilinear Transformations

Does normalization help? Games & Lucas (1966): Normalizing transformations hurt –Reduce interpretability, power Levin & Dunlap (1982): Transformations help –Increase power Games (1983): It Depends, Levin and Dunlap are stupid Levine & Dunlap (1984): It depends, Games is stupid Games (1984): This debate is stupid

Does non-normality hurt?

Normalize If and Only If It matters –In theory: Got robust? –In practice: Got change? Must assume normality (i.e., no non-parametric test available)

Missing Data

Why are they missing? –MCAR Variable’s missingness unrelated to both its value and other variables’ values e.g., equipment malfunction No bias –MAR Variable’s missingness unrelated to its value after controlling for its relation to other variables e.g., depression and income Bias –MNAR Variable’s missingness related to its value after controlling for its relation to other variables e.g., income reporting Bias

Diagnosing Missing Data How much? How concentrated? How essential? MCAR, MAR, MNAR? How influential?

Dealing with Missing Data –Treat missing data as data –Note bias “lower income individuals are underrepresented” –Delete variables –Delete cases Listwise Casewise –Estimation Prior knowledge Mean substitution Regression substitution Expectation-maximization (EM) Hot decking Multiple imputation (MI)

Missing Data: Conclusions Avoid missing data! If rare (<5%), MCAR, nonessential, concentrated, or impotent, delete appropriately If frequent, patterned, essential, diffuse, influential, use MI If MNAR, treat missingness as DV

Question: What’s the best method for identifying and removing RT outliers? Alternatives –RT cutoff (5 values) –z-score cutoff (1, 1.5) –Transformation (log, inverse) –Trimming –Medians –Windsorizing (2 SD)

Method Conduct series of simulations –DV: power (# sig simulations/1000) 2 x 2 ANOVA –One main effect (20, 30, 40 ms) 7 observations/condition –10% outlier probability –Outliers ms 32 participants Between-participants variability

SpreadDrift ex-Gaussian distribution

Inferences Absolute cutoffs resulted in greatest power Best cutoff values depended on type of effect –Shift: 10-15% cutoff –Spread: 5% cutoff Inverse transformation good, too With high between-participant variability, SD cutoff becomes effective

Recommendations Try range of cutoffs to examine robustness Replicate with inverse transformation (or SD cutoff) Replicate novel, unexpected, or important effects Choose method before analyzing data