1. 1 Reproducibility Working Group Stan Young Val Johnson Steve Goodman Stan Young

2. 2 Ioannidis Wakeup Call Randomized studies replicate ~80%. Non-randomized studies replicate ~20% (1/6).

3. 3 Vitamin C and the Common Cold At least 16 well-designed, double-blind studies have shown that supplementation with vitamin C does not prevent colds and at best may slightly reduce the symptoms of a cold.

4. 4 Vitamin E Fails to Deliver on Early Promise They took a supplement of up to 400 international units of the vitamin every day seemed to go on to develop fewer cases of heart disease or cancer than their peers who were not taking the supplement. As part of Harvard's double-blind, placebo-controlled Women's Health Study, nearly 20,000 healthy, middle-aged women were given 600 international units of vitamin E every other day for roughly 10 years. Results reported in the July 6 JAMA found no overall protective benefit against those illnesses.

5. 5 Low-Fat Diet Does Not Cut Health Risks, Study Finds The largest study ever to ask whether a low-fat diet reduces the risk of getting cancer or heart disease has found that the diet has no effect. The $415 million federal study involved nearly 49,000 women ages 50 to 79 who were followed for eight years. In the end, those assigned to a low-fat diet had the same rates of breast cancer, colon cancer, heart attacks and strokes as those who ate whatever they pleased, researchers are reporting today.

6. 6 Ioannidis Wakeup Call Randomized studies replicate ~80%. Non-randomized studies replicate ~20% (1/6). I informally count 19 claims coming from non-randomized studies and only one was confirmed in a randomized clinical trial, 94% failure rate. There is a shocking failure to replicate.

7. 7 Cause Effect Diagram

8. 8 Links to Working Groups Multiplicity Methods/References Teaching materials/review papers Subgroup Data sets – ACE, ? Methods / Review papers Replicability Subject areas

9. 9 Non-Statistical PR Groups ASA - biopharm Science writer Judicial Education ASA – Epi Popular articles Teaching materials OpEd

10. 10 Collect Literature Ioannidis 1 and 2 refs Randomized / Epi paper pairs Review papers

11. 11 Potential Papers 1.JAMA/NEJM Repro study 2.Stat Education – Teaching materials 3.TAS – Examples/causes 4.Stat Science – Stat / Epi 5.How to identify possible false positive results 6.Letters to editor 7.OpEd – TAS, Science, Newspapers 8.Costly examples 9.How to for false positive paper

12. 12 Sharing Data Sets National Academy book on data sharing. Journal, NIH and FDA policies Ask for data sets, Medical, Epi, etc. National data repository Data analysis competition

13. 13 Two Questions 1.Is your data set available? (Ask under what conditions.) 2. How many questions did you ask of this data set and did you adjust your analysis to reflect the number of questions?

14. 14 Teaching Materials

15. 15 www.niss.org/PowerArray Micro array, start to finish 20k lines of interface 300k lines of algorithms n<<p software 140 page user manual

16. 16 Dont fool yourself, or the customer. Stan Young NISS As a consumer, dont be fooled either!