RIMI Workshop: Power Analysis Ronald D. Yockey

Goals of the Power Analysis Workshop 1. Understand what power is and why power analyses are important in conducting research. 2. Recognize the limits of Null Hypothesis Significance Testing (NHST) and how effect sizes complement NHST. 3. Understand the relationship between power, effect size, and sample size. 4. Use GPower to estimate the sample size (N) required to obtain a desired level of power (e.g., 80%) for a number of statistical procedures. 5. Provide an estimate of power for your grant proposals!

Null Hypothesis Significance Testing (NHST)

What is Power? Power – the probability of rejecting the null hypothesis (i.e., obtaining significance) when it is false - Ranges from 0 to 1 - Ranges from 0 to 1 - When multiplied by 100%, power is expressed as a percentage

Examples of Power Example #1 Power = % of the time the null hypothesis will be rejected (i.e., statistical significance will be obtained) - 50% of the time the null hypothesis will not be rejected (i.e., statistical significance will not be obtained) – A Type II Error

Examples of Power (continued) Example #2 Power = % of the time the null hypothesis will be rejected (i.e., statistical significance will be obtained) - 20% of the time the null hypothesis will not be rejected (i.e., statistical significance will not be obtained) – A Type II Error

Rationale for Power Analysis - Work involved in a study - conceiving the idea, literature review, grant proposal submission, running participants, analyzing data, writing the results - High power = high chance of obtaining significance (supporting the research hypothesis) - Low power = low chance of obtaining significance - Neglecting “A priori” Power Analysis frequently results in low power studies - Power Analysis - Crucial for increasing the probability of getting significant results!

Rationale for Power Analysis (continued) Low power studies are very common e.g., Power = % chance of achieving significance (rejecting H 0 ) Is spending the time and effort to conduct the study (not to mention taxpayers’ money) worth it when there is only a 3 in 10 chance of getting significance? Recommended power level – 70% to 80% (Diminishing returns in the 90%+ range)

Factors That Influence Power 1. Alpha level (α =.05 or.01) - Larger α = greater power 2. One-tailed vs. two-tailed tests - One-tailed tests have greater power (for a constant α) - Two-tailed tests are much more common (a one- tailed test may require justification) 3. The size of the standard deviation (σ) - Smaller standard deviation = greater power (σ can be very difficult to manipulate)

Factors That Influence Power (continued) 4. Effect size – the size of the “treatment effect” in your study - Larger effect size = greater power 5. Sample size (N) - Larger N = greater power (The most commonly manipulated factor for increasing power)

Examples of Low Power Studies Very “realistic” low power study examples (for the independent samples t test): Example #1 (2-tailed, α=.05) n 1 = 30 n 2 = 30 Small effect (i.e., a relatively small difference between the groups; characteristic of many studies in the social and behavioral sciences) Power = 12%!

Examples of Low Power Studies (continued) Example #2 (2-tailed, α=.05) n 1 =50, n 2 =50; Small effect Power = 17%! Example #3 (2-tailed, α=.05) n 1 =30, n 2 =30; Medium effect Power = 48% All three studies suffer from insufficient power.

Rationale for Power Analysis (continued) The prevalence of low power studies is one reason why funding agencies such as NIH and NIMH (among others) often require estimates of power with the submission of a grant proposal. And that’s why we’re here today!

Null Hypothesis Significance Testing (NHST)

NHST (Continued) If statistical significance is obtained (e.g., p <.05), then we can declare that the groups are different. While a “statistically significant” result with NHST tells us the groups are different, it says nothing about how different they are. Statistical significance means “beyond normal sampling error” or “reliable difference,” but it does not necessarily mean “big difference” or “important.”

NHST (Continued) - While NHST can be a very useful tool, it has frequently been misused, as far too many researchers have made the mistake of assuming statistical significance means “practical importance” - Due to this common misunderstanding, the American Psychological Association (APA) now strongly encourages that effect sizes be presented (alongside the results of significance tests), and many journals require the reporting of effect sizes for manuscript consideration.

What is an Effect Size? Effect size – Indicates the size or degree of the effect of some treatment or phenomenon Definitions of effect size provided by Cohen (1988; p. 8-9) - “The degree to which the phenomenon is present in the population.” - “The degree to which the null hypothesis is false.”

NHST vs. Effect Size Cohen’s second definition of effect size (repeated): - “The degree to which the null hypothesis is false.” 1. NHST – If reject null – what do you conclude? The null is false – i.e., Experimental ≠ Control (NHST doesn’t indicate how different the groups are, just that they’re not equal) 2. Effect size – indicates how different the groups are

NHST vs. Effect Size (continued) Basic Question of Significance Testing (NHST) – Is there an effect? - Yes or No Basic Question of Effect sizes – How big is the effect? - A question of degree

Effect Sizes in Power Analysis Effect sizes play a fundamental role in power analysis – To conduct a power analysis, the effect size must be estimated. (We’ll examine several effect size measures shortly.)

Effect Sizes in Power Analysis (continued) Different effect sizes are often used for different statistical procedures (t tests, ANOVA, Correlation, etc.)

Effect Sizes – Mean Differences Effect size of the difference between two means Example #1 – IQ scores: group 1 = 115, group 2 = 105 Effect size = mean group 1 – mean group 2 = = 10 IQ points = = 10 IQ points Effect size of 10 IQ points (notice the effect size indicates how different the groups are)

Effect Sizes – Mean Differences (continued) Example #2: Stress – breathing exercises vs. control breathing exercises = 60, control = 67 breathing exercises = 60, control = 67 (higher scores = greater stress) Effect size = 60 – 67 = –7; effect size of 7 points (Often the absolute value for an effect size is reported.)

Effect Sizes – Mean Differences (continued) Problems with mean difference approach: 1. When different scales are used (with different M and SD) to measure the same construct, the results of different studies cannot be meaningfully compared (comparing apples and oranges). 2. Power analysis requires a standardized or “scale free” measure of effect size.

Standardized Measures of Effect Size t tests – Cohen’s d ANOVA – η 2 (eta-square) or R 2 Correlation – Pearson’s r Multiple Regression – R 2 Chi–Square Test of Independence – Cramer’s Phi

Cohen’s d Used for all t tests (one sample t, independent samples t, dependent samples t) A standardized or “scale free” measure of mean differences

Cohen’s d (continued)

Example: Examining the effect of a drug on pain levels - Pain questionnaire on a scale administered to people suffering from back pain (higher score = greater pain). - old drug – 25, new drug – 20 - standard deviation of 10.

Cohen’s d (continued) d =.5 (Interpret in terms of standard deviation differences - like z-scores) Those who took the new drug had pain levels that were.5 standard deviations lower than those who took the old drug.

Cohen’s conventions for d Magnituded Small.20 1/5 of a std. dev. difference Medium.50 1/2 of a std. dev. difference Large.80 8/10 of a std. dev. difference Cohen’s standards for small, medium, and large effect sizes for the independent samples t test, one sample t test, and the dependent samples t test.

Effect Size (d) Power.20 – Small.50 – Medium.80 – Large (388) 32 (64) 14 (28) (492) 41 (82) 17 (34) (620) 51 (102) 21 (42) (788) 64 (128) 26 (52) Sample size required per group (with total N listed in parentheses) for a given level of power and effect size for the Independent Samples t test (α =.05, 2-tailed). Note: Assumes equal n per group. Power Table – Independent t (abridged)

Cohen’s conventions for Pearson’s r Magnituder Small.10 Medium.30 Large.50 Cohen’s standards for small, medium, and large effect sizes for the Pearson r correlation coefficient.

Effect Size (r) Power.10 – Small.30 – Medium.50 – Large Sample size (N) required for a given level of power and effect size for the Pearson r correlation coefficient (α =.05, 2-tailed). Power Table – Pearson’s r (abridged)

Cohen’s Conventions for Cramer’s Phi/w (Chi-Square) Magnitude Phi, w Small.10 Medium.30 Large.50 Cohen’s standards for small, medium, and large effect sizes for the chi-square test of independence. Note: Applies only to 2 x k tables, where k ≥ 2.

Effect Size (Phi, w) Power.10 – Small.30 – Medium.50 – Large Sample size required for a given level of power and effect size for the chi-square test of independence (α =.05, df = 1, i.e., 2 x 2 table). Power Table – Chi-Square Test of Independence (abridged)

Effect Size - ANOVA k = the number of groups, m i = the mean of the ith group, m = the grand (overall) mean, and σ = the average (or pooled) standard deviation.

Effect Size - ANOVA

Cohen’s Conventions for ANOVA (f and η 2 ) Magnitudef η2η2η2η2 Small Medium Large Cohen’s standards for small, medium, and large effect sizes for the one-way between subjects analysis of variance (ANOVA).

Effect Size (f, η 2 ) Power f =.10; η 2 =.01 Small f =.25; η 2 =.06 Medium f =.40; η 2 =.14 Large (501) 28 (84) 12 (36) (627) 35 (105) 14 (42) (774) 43 (129) 18 (54) (969) 53 (159) 22 (66) Sample size (N) required per group (and total N) for a given level of power and effect size for the one-way between subjects ANOVA (α =.05). The power values provided are based on 3 groups; larger N is required to achieve the same level of power as the number of groups increase. Power Table – ANOVA (abridged)

Effect Size – Multiple Regression

Cohen’s conventions for Multiple Regression (f 2 and R 2 ) Magnitude f2f2f2f2 R2R2R2R2 Small Medium Large Cohen’s standards for small, medium, and large effect sizes for multiple regression.

Effect Size (f 2, R 2 ) Power f 2 =.02; R 2 =.02 Small f 2 =.15; R 2 =.13 Medium f 2 =.35; R 2 =.26 Large Sample size (N) required for a given level of power and effect size for multiple regression (α =.05). The power values provided are based on 3 predictors (IVs); larger N is required to achieve the same level of power as the number of predictors increase. Power Table – Multiple Regression ( abridged)

Estimating Power using GPower GPower illustration…