Copyright © 2011 by Pearson Education, Inc. All rights reserved Statistics for the Behavioral and Social Sciences: A Brief Course Fifth Edition Arthur Aron, Elaine N. Aron, Elliot Coups Prepared by: Genna Hymowitz Stony Brook University This multimedia product and its contents are protected under copyright law. The following are prohibited by law: -any public performance or display, including transmission of any image over a network; -preparation of any derivative work, including the extraction, in whole or in part, of any images; -any rental, lease, or lending of the program.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Making Sense of Statistical Significance Chapter 7

Copyright © 2011 by Pearson Education, Inc. All rights reserved Chapter Outline Effect Size Statistical Power What Determines the Power of a Study? The Role of Power When Planning a Study The Role of Power When Interpreting the Results of a Study Effect Size and Power in Research Articles

Copyright © 2011 by Pearson Education, Inc. All rights reserved Effect Size An effect can be statistically significant without having much practical significance. Effect Size –It is a measure of the difference between populations. –It tells us how much something changes after a specific intervention. –It indicates the extent to which two populations do not overlap. how much populations are separated due to the experimental procedure –With a smaller effect size, the populations will overlap more.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Figuring The Effect Size Raw Score Effect Size –calculated by taking the difference between the Population 1 mean and the Population 2 mean Standardized Effect Size –calculated by dividing the raw score effect size for each study by each studys population standard deviation

Copyright © 2011 by Pearson Education, Inc. All rights reserved Effect Size Example If Population 1 had a mean of 90, Population 2 had a mean of 50, and the population standard deviation was 20, the effect size would be: –(90 – 50) / 20 = 2 This indicates that the effect of the experimental manipulation (e.g., reading program) is to increase the scores (e.g., reading level) by 2 standard deviations.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Formula for Calculating the Effect Size Effect Size = Population 1 M – Population 2 M Population SD –Population 1 M = the mean for the population that receives the experimental manipulation –Population 2 M = the mean of the known population (the basis for the comparison distribution) –Population SD = the standard deviation of the population of individuals –A negative effect size would mean that the mean of Population 1 is lower than the mean of Population 2.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Example of Calculating the Effect Size For the sample of 64 fifth graders, the best estimate of the Population 1 mean is the sample mean of 220. The mean of Population 2 = 200 and the standard deviation is 48. Effect Size = Population 1 M – Population 2 M Population SD Effect Size = 220 – Effect Size =.42

Copyright © 2011 by Pearson Education, Inc. All rights reserved Effect Size Conventions Standard rules about what to consider a small, medium, and large effect size –based on what is typical in behavioral and social science research Cohens effect size conventions for mean differences: –small effect size =.20 –medium effect size =.50 –large effect size =.80

Copyright © 2011 by Pearson Education, Inc. All rights reserved A More General Importance of Effect Size Knowing the effect size of a study lets you compare results with effect sizes found in other studies, even when the other studies have different population standard deviations. –Knowing the effect size lets you compare studies using different measures, even if the measures have different means and variances. Knowing what is a small or a large effect size helps you evaluate the overall importance of a result. –A result may be statistically significant without having a very large effect. Meta-Analysis –a procedure that combines results from different studies, even results using different methods or measurements –This is a quantitative rather than a qualitative review of the literature. –Effect sizes are a crucial part of this procedure.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Statistical Power Probability that the study will produce a statistically significant result if the research hypothesis is true –When a study has only a small chance of being significant even if the research hypothesis is true, the study has low power. –When a study has a high chance of being significant when the study hypothesis is actually true, the study has high power. Statistical power helps you determine how many participants you need. Understanding power helps you make sense of the results that are not significant or results that are statistically significant but not of practical importance. To determine power, researchers can use a power software package or a power calculator. Power can also be found on a power table.

Copyright © 2011 by Pearson Education, Inc. All rights reserved How Are You Doing? What is effect size? How is effect size calculated? Why is effect size important? What is statistical power? Why is power important?

Copyright © 2011 by Pearson Education, Inc. All rights reserved What Determines the Power of a Study? The statistical power of a study depends on: –how big an effect the research hypothesis predicts effect size –how many participants are in the study sample size –other factors that influence power include: significance level chosen whether a one-tailed or two-tailed test is used the kind of hypothesis-testing procedure used

Copyright © 2011 by Pearson Education, Inc. All rights reserved Effect Size and Power If there is a is a mean difference in the population, you have more chance of getting a significant result in the study. –If you predict a bigger mean difference, the power based on that prediction will be greater. –Since the difference between population means is the main component of effect size, the bigger the effect size, the greater the power. –Effect size is also determined by the standard deviation of a population. The smaller the standard deviation, the bigger the effect size. –The smaller the standard deviation, the greater the power.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Sample Size The more people there are in the study, the greater the power is. The larger the sample size, the smaller the standard deviation of the distribution of means becomes. –The smaller the standard deviation of the distribution of means, the narrower the distribution of meansand the less overlap there is between distributions leading to higher power. Remember that though sample size and effect size both influence power, they have nothing to do with each other.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Figuring Needed Sample Size for a Given Level of Power The main reason researchers consider power is to help them decide how many people to include in their studies. –Sample size has an important influence on power. –Researchers need to ensure that they have enough people in the study that they will be able see an effect if there is one.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Other Influences on Power Significance Level –Less extreme significance levels (e.g., p <.10) mean more power because the shaded rejection area of the lower curve is bigger and more of the area in the upper curve is shaded. –More extreme significance levels (e.g., p <.001) mean less power because the shaded region in the lower curve is smaller. One- vs. Two-Tailed Tests –Using a two-tailed test makes it harder to get significance on any one tail. Power is less with a two-tailed test than a one-tailed test.

Copyright © 2011 by Pearson Education, Inc. All rights reserved The Role of Power When Planning a Study Even if the research hypothesis is true, if you conduct a study that has low power, your results will most likely not be statistically significant. There are some practical ways a researcher can increase the power of a study. –Increase the effect size by increasing the predicted difference between population means. Use a more intense experimental procedure. Elaborate on the instructions in the experimental condition. Methods to increase the impact of an experimental procedure can be difficult or costly and may lead you to use a procedure that is very different from the situation to which you would like to generalize your results. –Increase the effect size by decreasing the population standard deviation. Use a population that has less variation within it than the one you planned to sample. –The disadvantage to this is that the results will then only apply to the more limited population. Use more standardized testing conditions and more precise measures. –Test in a controlled laboratory setting. –Use measures and tests with very clear wording. –Increase the sample size. This is the main way to change a planned study to raise its power. There is a limit to how many participants are available. A larger sample adds to the time and cost of conducting research. –Use a less extreme level of significance. Using a level of significance that is less extreme than p <.05 is not recommended because this increases the chance of making a Type I error. –Use a one-tailed test. It is rarely possible to change this, as it is dependent on the logic of the hypothesis being studied. –Use a more sensitive hypothesis-testing procedure. It is rarely possible to change this, as researchers generally attempt to use the most sensitive hypothesis- testing procedure available to them.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Statistical Significance vs. Practical Significance –It is possible for a study with a small effect size to be significant. Though the results are statistically significant, they may not have any practical significance. –e.g., if you tested a psychological treatment and your result is not big enough to make a difference that matters when treating patients Evaluating the practical significance of study results is important when studying hypotheses that have practical implications. –e.g., whether a therapy treatment works, whether a particular math tutoring program actually helps to improve math skills, or whether sending mailing reminders increases the number of people who respond to the Census With a small sample size, if a result is statistically significant, it is likely to be practically significant. In a study with a large sample size, the effect size should also be considered.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Role of Power When a Result is Not Statistically Significant A nonsignificant result from a study with low power is truly inconclusive. A nonsignificant result from a study with high power suggests that: –the research hypothesis is false or –there is less of an effect than was predicted when calculating power

Copyright © 2011 by Pearson Education, Inc. All rights reserved Role of Significance in Sample Size When Interpreting Research Results If the result is statistically significant and the sample size is small, the result is important. If the result is statistically significant and the sample size is large, the result might or might not have practical implications. If the result is not statistically significant and the sample size is small, the result is inconclusive. If the result is not statistically significant and the sample size is large, the research hypothesis is probably false.

Copyright © 2011 by Pearson Education, Inc. All rights reserved How Are You Doing? What are some practical ways of increasing the power of a research study? What is the difference between statistical significance and practical significance?

Copyright © 2011 by Pearson Education, Inc. All rights reserved Effect Size and Power in Research Articles Articles often mention effect size. Effect size is a crucial factor in meta- analyses, and thus is almost always reported in meta-analyses. Power is sometimes discussed when evaluating nonsignificant results.

Copyright © 2011 by Pearson Education, Inc. All rights reserved Key Points Effect size is a measure of the difference between population means. It is figured by dividing the difference between population means by the population standard deviation. Small effect size =.20, Medium =.50, Large =.80 Statistical power is the probability that the study will produce a statistically significant result if the research hypothesis is true. The larger the effect size, the greater the power. The larger the sample size, the greater the power. Power is also affected by significance level, whether a one- or two- tailed test is used, and the type of hypothesis-testing procedure. Statistically significant results from a study with high power may not have practical importance. Non-significant results from a study with low power are inconclusive. Research articles often report effect size, and effect sizes are always discussed in meta-analyses. Power is sometimes discussed in research articles.