PSY 1950 Confidence and Power December,

Requisite Quote “The picturing of data allows us to be sensitive not only to the multiple hypotheses that we hold, but to the many more we have not yet thought of, regard as unlikely, or think impossible.” –Tukey, 1974

Point Estimation vs. Interval Estimation Confidence intervals estimate parameters with intervals instead of simply points

Reliability Confidence intervals indicate the reliability of an estimate

Confidence Level Confidence intervals are a function of the desired precision

Confidence Interval Definition –“For a given proportion p (where p is the confidence level), a confidence interval for a population parameter is an interval that is calculated from a random sample of an underlying population such that, if the sampling was repeated numerous times and the confidence interval recalculated from each sample according to the same method, a proportion p of the confidence intervals would contain the population parameter in question”

Correct Interpretation e.g., “If we replicated our experiment, the calculated confidence intervals would contain the true population mean 95% of the time.” e.g., “The confidence interval represents values for the population parameter for which the difference between the parameter and the observed estimate is not statistically significant at the 5% level."

Incorrect Interpretation Confidence intervals do NOT reflect the probability that the parameter falls within an estimated range –Wrong: “There is a 95% chance that the actual mean group difference is between 2-3.” The true mean is fixed, not variable: it either falls inside or outside a particular range –e.g., what is the probability that 2 is between 3- 4?

“This is not just quibble” Imagine CIs constructed from two samples from the same population –1st sample’s 70% CI: % chance that the mean is between 1-2 –2nd sample’s 70% CI: % chance that the mean is less than 2 or greater than 3 –These are incompatible!

Eyeballing Interval Estimates Question matters –Do you care about estimated values or differences of estimated values? Values matters –Note actual values of point estimates (and relevant differences) –Note actual values of interval estimates (and relevant differences) –SE, SD, CI? Context matters –How you interpret CI on individual group/condition means depends on experimental design

Independent Measures Difference CI can be inferred from mean CIs –Always larger –If group CIs are similar, larger by factor of √2

Dependent Measures Difference CI cannot be inferred from mean CIs –Depends on correlation –Usually smaller than mean CIs (r is usually positive and large)

Rules of Thumb: 95% CI So long as ns are at least 10 and one error is not greater than twice the other: –If proportion overlap is.5 or less, t-test is significant at  =.05 –If there is no overlap, t-test is significant at  =.01

Rules of Thumb: SE So long as ns are at least 10 and one error is not greater than twice the other: –If proportion gap is 1 or greater, t-test is significant at  =.05 –If proportion gap is 2 or greater, t-test is significant at  =.01

CI Calculation

Example

Dependent Measures

Additional Resources

CIs and Replication Given a sample mean and 95% CI, what is the probability that a repetition of that experiment, with an independent sample of participants, would give a mean that falls within the original CI? –Not.95 –Actually.834 Why? –Probability that CI “captures” population mean depends on: Deviation of sampled mean from population mean –Probability that CI “captures” replicant mean depends on: Deviation of original (sample) mean from population mean Deviation of replicant (sample) mean from population mean

Power The probability of correctly rejecting a false null hypothesis The probability of not making a Type II error Power depends on –Effect size –Sample size –Alpha Significance = Effect size  Sample size

Prospective Power Analysis Question: How many subjects do I need to have reasonable odds of getting a significant effect? Calculation –Determine desired power Usually.8 –Estimate effect size Prior research Theoretical considerations –Solve for n

Retrospective Power Analysis Question: If the true effect size was the one I found, what was the power of my experiment? Calculation –Use obtained effect size and sample size Controversial/wrong claim: If you had high retrospective power and nonsignificant results, the null hypothesis is probably true –It is not possible to have high retrospective power and reject the null With p =.05, power =.5 With p >.05, power <.5