Q-Vals (and False Discovery Rates) Made Easy Dennis Shasha Based on the paper "Statistical significance for genomewide studies" by John Storey and Robert Tibshirani PNAS August 5,

Challenge You test plants/patients/… in two settings (or from different populations). You want to know which genes are differentially expressed (alternate) You don’t want to make too many mistakes (declaring a gene to be alternate when in fact it’s null – not differentially expressed).

First Idea You take p-vals of the differences in expression. P-val(g) is the probability that if g is null, it would have a difference at least this large. You choose a cutoff, say You say all genes that differ with p-val <= 0.05 are truly different. What’s the problem?

Thought Experiment Suppose that no genes are truly differentially expressed. You will conclude that about 5% of those you called significant really are. Your false discovery rate (number null among those predicted to be alternate/number predicted to be alternate) = 100%. Bad.

A Fundamental Insight All truly null genes (i.e. not truly differentially expressed) are equally likely to have any p-val. That is by construction of p-val: under the null hypothesis, 1% of the genes will be in the top 1 percentile, 1% will be in percentile between 89 and 90 th and so on. P-val is just a way of saying percentile in null condition.

What Do We Do With That? Mixture model: imagine null genes as light blue marbles and truly different genes as red ones. If the assay is decent, red marbles should be concentrated at the low p-values.

0 …. Pval …………………………………………………1

Method We Can Use We don’t of course know the colors of the marbles/we don’t know which genes are true alternates. However, we know that null marbles are equally likely to have any p-value. So, at the p-value where the height of the marbles levels off, we have primarily light blue marbles/null genes. Why?

0 …. Pval …………………………………………………1 Flat region starts here Level of flat region

Answer Because if all genes/marbles were null, the heights would be about uniform. Provided the reds are concentrated near the low p-vals, the flat regions will be primarily light blues.

Example: all null Consider the all null case. All marbles are light blue. False discovery rate in region to left of flat region is estimated number of white marbles (based on flat region)/number of marbles to left of flat region. This will be close to 100%

0 …. Pval …………………………………………………1 Flat region starts here Level of flat region

Example: all non-null Consider the all non-null case. All marbles are red and they are highly skewed. Flat region is essentially zero. False discovery rate in region to left of flat region is estimated number of white marbles (based on flat region)/number of marbles to left of flat region. This will be close to 0.

0 …. Pval …………………………………………………1 Flat region starts here

Example: mixed case Get a distribution of p-values. Find flat region. Estimate number of nulls in the left-of-flat region by extending the flat line. This gives the false discovery rate.

0 …. Pval ……………………………………………1 Flat line; base level of nulls Number of genes having pval Possible p-value threshold

Example: mixed case What would you estimate the false discovery rate to be in the case that we declare the entire area to the left of the possible p-value threshold to be significant? 10%, 25%, 50%?

0 …. Pval ……………………………………………1 Flat line; base level of nulls Number of genes having pval Possible p-value threshold

Obtaining q-values from False Discovery Rate Suppose we order genes from least p- value to greatest. That corresponds to one of these cartesian graphs. The q-value of a gene having p-value p is exactly the False Discovery Rate if the declared significance region had a threshold of p.

0 …. Pval ……………………………………………1 Flat line; base level of nulls Number of genes having pval Q-value of a gene having this p-val is the FDR if this is the significance threshold.

Lessons for Research Mushy p-values (large error bars/few replicates) may force us to the far left in order to get a low False Discovery Rate. This may eliminate genes of interest. If testing out a gene is not too expensive, then we can accept a higher False Discovery Rate – nothing magical about 0.01.

0 …. Pval ……………………………………………1 Flat line; base level of nulls Number of genes having pval Better p-values avoid loss of genes, for small FalseDiscovery Rate.