Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014

Overview Discussion of the Hedges / Hanushek exchange. Investigation of vote counting as an inferential method.

Education finance exchange Larry Hedges was then a professor of educational statistics at the University of Chicago. Eric Hanushek was an economist at the University of Rochester. Now at Northwestern and Stanford, respectively.

Background of the paper Hanushek at the time was extremely active as an expert witness in educational equity lawsuits. Paper grew out of a student project in Hedges’ meta-analysis class. Discussion.

Vote counting as inference To understand vote counting as an inferential method, we need to understand the probability that an individual study will reject the null hypothesis. Statisticians have a name for that idea. Power.

What does power depend on? Lots of things: –characteristics of population –choices about how to do inference –characteristics of the sample.

Characteristics of the population How strong is the effect? How much unmodeled variability exists?

Choices about how to do inference Alpha level. One- vs. two-tailed tests.

Characteristics of the sample Sample size.

Back to vote counting To understand vote counting, we need to understand power. We’ve just seen that power is a complex function of lots of factors. If we want to understand something that is too complex, what can we do? Simplify.

Simplifying All of those issues that were characteristics of the population can be simplified by, for the moment, confining our interests to the fixed-effects context. In that case, we are assuming that all of the studies are samples from the same population.

Simplifying All of the issues that were characteristics of how we do inference are under our control. For example, we can simply say that a vote is positive if the null hypothesis is rejected using a two-tailed test with an alpha level of.05.

Simplifying The remaining issue that effects power is the sample size of the individual study. Obviously, in the real world, different studies will have different N. Simplify by assuming that all studies have the same N.

Simplifying With these simplifying assumptions, we can treat power (i.e., the probability of a positive vote) as a constant.

Distribution of votes We can now think of our studies as a series of independent attempts to vote YES. For each attempt, the probability of a YES vote is the same (power). We are interested in the total number of YES votes. This should sound vaguely familiar.

Distribution of votes Suppose that instead of studies and YES votes, I were talking about coin tosses and HEADS outcomes. We would be looking at a series of independent coin tosses with a constant probability of success, and would be interested in the probability of a particular number of successes.

Distribution of votes With the simplifying assumptions we have made, the number of YES votes follows a binomial distribution.

What should we assume for power? Given that Hanushek is arguing that there is no effect, we should be justified in considering it to be “small.” Empirical studies of power.

Cohen 1962 The statistical power of abnormal-social psychological research, Journal of Abnormal and Social Psychology, 65, Finding: 100% of studies of small effects in that field had power of <.50.

Brewer 1973 A note on the power of statistical tests in the Journal of Educational Measurement, Journal of Educational Measurement, 10, Very much the same finding as Cohen.

Understanding vote counting What happens with vote counting as the number of studies becomes large? (Another digression in R.) Using the normal approximation to the binomial distribution.