Power analysis

Common "general format" of most statistical tests An effect (presented as the variance it has produced) of the supposed relation is compared with the total variance in the sample

State of the World (~The truth) Negative (H O )Positive (H 1 ) Decision Negative (H 0 ) True negaive False negative (Type II Error  Positive (H 1 ) False positive (Type I Error  True positive

  : probability to miss the effect when it does exist  confidence level, probability to claim the effect presence when there is none 1-   power of the test R: ratio N ”effect” / N ”no effect” in the space of research findings

In most situations in statistical analysis, we do not have access to an entire statistical population of interest, either because the population is too large, is not willing to be measured, or the measurement process is too expensive or time-consuming to allow more than a small segment of the population to be observed. As a result, we often make important decisions about a statistical population on the basis of a relatively small amount of sample data. Typically, we take a sample and compute a quantity called a statistic in order to estimate some characteristic of a population called a parameter.

One thing is virtually certain before the study is ever performed: The population proportion (P) will not be equal to the sample proportion (p). Because the sample proportion (p) involves "the luck of the draw," it will deviate from the population proportion (P). The amount by which the sample proportion (p) is wrong, i.e., the amount by which it deviates from the population proportion (P), is called sampling error.

In any one sample, it is virtually certain there will be some sampling error (except in some highly unusual circumstances), and that we will never be certain exactly how large this error is. If we knew the amount of the sampling error, this would imply that we also knew the exact value of the parameter, in which case we would not need to be doing the opinion poll in the first place.

In general, the larger the sample size N, the smaller sampling error tends to be. (although one can never be sure what will happen in a particular experiment). If we are to make accurate decisions about parameter p, we need to have an N large enough so that sampling error will tend to be "reasonably small." If N is too small, there is not much point in gathering the data, because the results will tend to be too imprecise to be of much use.

Statistical power increases when: Sample larger; Statistical test better suited (fits the case); Signal (effect size, relationship) stronger; Noise (error, residual variance) weaker.

Ioannidis corollaries 1.Smaller studies (in terms of sample size); 2.Smaller effect sizes; 3.Greater number and the lesser selection (?) of tested relationships; 4.Greater flexibility in designs, definitions, outcomes, and analytical modes in a scientific field; 5.Greater financial and other interests and prejudices; 6.Hotter scientific field (with more scientific teams involved). The research findings are less likely to be true in the cases of:

Experimental design: think before the experiment If you haven’t done so, then: Not enough data points The influential factors were not controlled for. Hence: –The influence cannot be measured –Their impact cannot be subtracted Extra factors introduce more variance (= noise) The design has been intractable, thus cannot be analyzed

But still, if you have got to think later… Reduce the design: cancel options that won’t fit Apply a “still possible” method Randomize afterwards – if the sample size allows… All this should not disagree with the purpose of the study!

FunCoup: a meta-analysis heavily dependent on published research findings