Statistical Power 1

First: Effect Size The size of the distance between two means in standardized units (not inferential). A measure of the impact of an intervention based on the distributions of the samples in your study. We use two measures of effect size – Cohen’s d – Eta Squared (η 2 ) 2

Effect Size: Cohen’s d Cohen’s d equals the difference in the means divided by the average of the standard deviations. It describes the distance between the means in units of pooled standard deviation (remember z scores?). It is a standardized measure of the impact of a statistically significant intervention (independent variable). 3 X2X2 X1X1

Effect Size: Eta Squared (η 2 ) Eta squared is the ratio of between group variance (impact of the intervention) to total variance. As the between group variance becomes a larger portion of the total variance (more intervention impact), eta squared gets closer to 1. Eta squared is a standardized measure of the explanatory power of the independent variable. 4 Between group variance Total variance Within group variance sum of squares between sum of squares total η 2 =

Effect Size Labeldrr 2 or η 2 Extremely large effect Very large effect Large effect Medium effect Small effect Although all the measures of effect size represent the impact of the independent variable in somewhat different ways, they are equivalent. 5 For an expanded version see the Effect Size table on the website

Effect Size In the real world how much difference did the independent variable make? Effect size is not based on inference. It is based on observed measures. 6 X2X2 X1X1

Sample Mean Sample Distribution Sampling Distribution Mean for a Given Group Size Sampling Distribution for a Given Group Size Now: Inferential Mistakes

Avoiding Type II Errors.05 Sampling Distribution of the Mean Theoretical distribution based on randomly selected groups of a given size. Means in this area would appear randomly less than 5% of the time.

Avoiding Type II Errors.05 Sometimes a mean score would not be identified as significant because it is likely to appear randomly more than 5% of the time. Sometimes that mean score represents a real world change in what is being measured but the difference isn’t enough to be significant. Type II Error

Avoiding Type II Errors.05 Now a larger group size moves the point at which the alpha level appears and something that wasn’t significant becomes so. Using larger groups reduces type II errors..05

Avoiding Type II Errors.05 Great. Using larger group sizes helps reduce type II errors. But, the cost is that developing large samples is difficult. We need to figure out how big the sample size needs to be to reasonably reduce type II errors but still keep the group as small as possible.

Power Power is defined as the probability of finding significance if it exists (avoiding type II errors). Eighty percent (.80) is accepted as a reasonable target power. If non-random change occurs it has an 80% probability of being observed. 12

Power There are 2 ways to use power calculations. First, they can be used to figure out appropriate sample sizes for a study. Second, they can be used to evaluate the use of a specific sample size after a study has been completed. 13

Power and Effect Size If a study shows larger effect sizes, smaller sample sizes will still be expected to show significance. Conversely, smaller effect sizes would require larger sample sizes. Fortunately, all of this can be read off of a table. 14 X2X2 X1X1

Power and Effect Size 15 Choosing Sample Sizes When Designing a StudyUsing Power to Explain Results

While we are here … Remember we have talked about inferential errors when something appears significant but it really wasn’t? Type I errors 16

Avoiding Type I Errors.05 Every 100 times a mean appear in the.05 area, 5 of them would have occurred randomly. That means we would identify something as not random (significant) 5 times out of 100 and be wrong. Type I Error

Avoiding Type I Errors.01 Solution: Move the alpha level to.01 That means we would identify something as not random (significant) 1 time out of 100 and be wrong. Less chance of a Type I Error

Avoiding Type I Errors.05 But this is social science and there is no good reason to make it this difficult to demonstrate significance..05 is a reasonable alpha level.

Avoiding Type I Errors.05 With larger group sizes a given point moves to a smaller probability of appearing. Using larger groups reduces type I errors..05

Sample Sizes We know that using larger sample sizes is statistically powerful but life isn’t that simple. Whenever possible use Power Analysis to help you be more confident you will find something if it is there. When things don’t appear to be significant at least now Power Analysis gives you something else to talk about to suggest what might be done to improve the quality of your data. 21

Examples 1.Most of the studies in your lit review are showing medium effects around 0.4 Cohen’s d. You want to be 90% sure you find non-random effects if they are there. Approximately how big does your sample need to be? 2.In your study you showed mean differences of 0.4 Cohen’s d but groups were not significantly different. Your sample size was 30. What was the probability of finding significant differences if they were there? 22

Analysis

Inferential Statistics Assumptions – Dependent variable is an interval measure of one characteristic of a group. – Tests are based on knowing or assuming the distribution of a population. – Statistics demonstrate if comparison samples are from the same population.

Testing Group Differences IndependentDependentTest 1 (1 group)1 measure 2 times (intervention in between)Paired t-test 1 (2 groups)1 measure (usually after the intervention) Independent samples t-test 1 (1 group) 1 measure 3 or more times (usually two after the intervention) Repeated measures ANOVA 1 (2 or more groups) 1 measure (usually after the intervention) Single factor ANOVA Non-Parametric 1 group2 non-interval measuresChi-square EZA

Post Hoc Tests Test ANOVASimilar group sizesTukey’s ANOVADissimilar group sizesScheffe’s EZA

Practical Significance (not inferential) Test Pooled Standard DeviationCohen’s d Ratio of variancesEta Squared EZA

Testing Group Differences (Things We Haven’t Done) IndependentDependentTest 2 (2 or more groups) 1 measure Factorial (2-way ANOVA) Shows interaction between groups 2 (2 or more groups) 1 measure 2 or more times ANCOVA (Analysis of Co-Variance) Allows for control of instance of dependent measure 2 or more variables 2 or more measures MANOVA (Multiple Analysis of Variance) 2 or more variables 2 or more measures 2 or more times MANCOVA (Multiple Analysis of Co-Variance) Allows for control of instance of dependent measure. Non-Parametric 2 (2 or more groups) Non-intervalKruskal-Wallis (ranked sum) EZA?

Correlational Statistics Assumptions – The relationship among the measures of two characteristics is linear. – Compared measures come from individuals in the same population – Correlations are not causal

How Do Variables Relate? ComparisonTest Association of 2 or more interval measuresPearson’s r Association of 2 measures at least one of which is not interval (ranked comparisons) Spearman’s ρ (rho) Measure of internal consistencyCronbach’s alpha Prediction based on association of 2 measuresLinear Regression Things We Haven’t Done Association of 3 or more measures at least one of which is not interval (ranked comparisons) Kendall’s τ (tau) Prediction based on 3 or more associationsMultiple Regression Finding relationships among items in a set of items (data reduction) Factor Analysis EZA EZA? EZA