1. Education 795 Class Notes Data Analyst Pitfalls Difference Scores Effects Sizes Note set 12

2. Today’s Agenda Announcements (ours and yours) Data Analyst Pitfalls Difference Scores Effect Sizes Multiple Comparisons

3. Data Analyst We meet the qualitative paradigm and position ourselves in the research, quantitative discourse now includes how we the researcher affects the following: What is to be investigated How it is to be done What are the facts? Hypotheses? What are the findings? Intepretations?

4. Pitfalls Improper analyses used We leave out a lot of the details that enable others to properly evaluate the research Use rules that have no consensus in the field Some try to replicate with the same data to refute results… (we’ve seen this before) We often ignore Validity and it is actually the crux of what we conclude We given limited attention to Reliability

5. Difference Scores The most common difference score Calculate Pretest-Postest for each subject Attain mean difference scores for treatment and comparison groups Test the difference of the difference scores for statistical signficance Limitation This requires the pre and post test to have the same factor structure.

6. Problems with Difference Scores Often referred to as ‘gain scores’ Improvement ranges are often not equal for individuals along the continuum. Example: Room for improvement is greater for those that start lower on the pre-test scale. If there is a ceiling effect (too easy) or a floor effect (too hard) difference scores are meaningless because there will likely be no change Short time intervals often don’t allow change

7. Alternatives to Raw Gain Scores Standardized gain scores Residualized gain scores ANCOVA design predicting the post- test controlling for the pre-test Here we lose some valuable information about individual change

8. Effect Sizes Effect sizes are used to refer to magnitude, importance and meaningfulness. Cohen (1988) defined ES as “the degree to which the phenomenon is present in the population”. For Cohen’s d (specifically designed to asses the difference between groups), the rule is.2 small,.5 medium and.8 as large effects. For correlation coefficients,.1 small,.3 medium,.5 large effects

9. Connecting ES to Power Cohen (1962) showed that the median power to detect small, medium and large effects was.17,.46 and.89. In other words, for a small effect, holding sample sizes constant, a test will only correctly reject the null hypothesis 17% of the times For large effects present in the population, tests will correctly reject the null hypothesis 89% of the time. Note that large effects are rarely found in sociobehavioral research

10. Effect Sizes There are two major classes of effect sizes (not counting a third "miscellaneous" category described by Kirk (1996)): (a) variance-accounted-for effect sizes analogous to a squared correlation coefficient (b) standardized mean differences

11. Types of Effect Sizes Squared Multiple Correlation Coefficient We’ve seen this before, it is R 2 R 2 =Sum of Squares Regression (Effect) /Sum of Squares Total Percent of variance in the outcome explained Note there is no effect of sample size on this statistic. This statistic is similar to  2 or Eta squared for ANOVA. Note we can calculate  2 for each separate main effect in an ANOVA table.

12. Effect Sizes for Means 1. Cohen's d = M1 - M2 / s pooled where s pooled =  [((s 1 )²+ (s 2 )²) / 2] Example: Assume equal n’s. M1 = 50, M2 = 60, s1=10, s2=15 s pooled = sqrt((10 2 + 15 2 )/2) = sqrt(325/2) = sqrt(162.5) = 12.7 Cohen’s d=60-50/12.7 =.78 ---LARGE EFFECT

13. Effect Size for Means We can also calculate an effect size by the t-statistic. Cohen's d = 2t  (df) In 1999, the APA Task Force on Statistical Inference published its report: http://www.apa.org/journals/amp/amp5485 94.html http://www.apa.org/journals/amp/amp5485 94.html

14. Laptop Project In groups of 4 Run a Regression using at least one group membership variable. Use the descriptive statistics to calculate an effect size for the group, compare that to the statistical significance. Present your results to the class.