How Do We Improve Research Methods and Reporting? Shai D. Silberberg NINDS / NIH Disclaimer Opinions I will voice are not official opinions of NIH!  What are the problems?  What happens if we don’t improve research?  How can we fix the problems?

Prinz, Schlange and Asadullah Nature Reviews Drug Discovery 2011; 10: / 67 Begley and Ellis. Nature 2012, 483: “scientific findings were confirmed in only 6 (11%) cases.”

Confounding variables Clever Hans Problems with resources Science at the cutting edge What Are The Causes For Poor Reproducibility? Lack of transparency in reporting Deficient experimental procedures Publication bias Human nature Poor reproducibility ××=×

“Human action can be modified to some extent, but human nature cannot be changed.” Abraham Lincoln Cooper Union Address New York, New York February 27, 1860 Human Nature

“The moment one has offered an original explanation for a phenomenon which seems satisfactory, that moment affection for his intellectual child springs into existence......So soon as this parental affection takes possession of the mind, there is a rapid passage to the adoption of the theory. There is an unconscious selection and magnifying of the phenomena that fall into harmony with the theory and support it, and an unconscious neglect of those that fail of coincidence…. ….There springs up, also, an unconscious pressing of the theory to make it fit the facts, and a pressing of the facts to make them fit the theory.” Journal of Geology, 1897 Thomas Chrowder Chamberlin The Method of Multiple Working Hypotheses

Chalmers et al., N Engl J Med 1983; 309:

Schulz et al., PLOS Medicine 2010; 7: 1–7 “Randomized trials can yield biased results if they lack methodological rigour. To assess a trial accurately, readers of a published report need complete, clear, and transparent information on its methodology and findings.” The CONSORT statement provides guidelines for reporting clinical trials

Among the 35 items included in the CONSORT guidelines are:  How sample size was determined  Method used to generate the random allocation sequence  Mechanism used to implement the random allocation sequence  Who was blinded after assignment to interventions and how  Losses and exclusions after randomisation, together with reasons Ignoring any one of these items can lead to bias

Chavalarias and Ioannidis, J Clin Epid 2010; 63:

“The reliability of a study is determined by the investigator’s choices about critical details of research design and conduct” (David F. Ransohoff, J Clin Oncol 28: ) “Bias is unintentional and unconscious. It is defined broadly as the systematic erroneous association of some characteristic with a group in a way that distorts a comparison with another group…..” “…..The process of addressing bias involves making everything equal during the design, conduct and interpretation of a study, and reporting those steps in an explicit and transparent way.” The definition of bias

What about pre-clinical studies? Malcom MacleodEmily SenaDavid Howells

Sena et al., JCBFM. 2014; 34: Insufficient reporting of methodological approaches is evident for pre-clinical studies

Effect size for studies of FK506 (Tacrolimus) in experimental stroke. Sena et al., Trends Neurosci 2007; 30: The fewer methodological parameters are reported, the greater the apparent efficacy!

Hackam and Redelmeier, JAMA 2006; 14: Journals: Cell Nature Science Nature Medicine Nature Genetics Nature Immunology Nature Biotechnology >500 citations Inadequate reporting is widespread

“Peer review is the evaluation of work by one or more people of similar competence to the producers of the work.” Peer Review Wikipedia

Year Publications (x10 6 ) The Escalation in Scientific Reporting (Annual PubMed Publications in English)

Publish or perish!Grant support Impact factorInnovation SignificanceNovelty

Hackam and Redelmeier, JAMA 2006; 14: Journals: Cell Nature Science Nature Medicine Nature Genetics Nature Immunology Nature Biotechnology >500 citations Inadequate reporting is widespread

Amyotrophic lateral sclerosis (ALS)  Death within 5 years of diagnosis  Central pathological finding is motor neuron death  3% of cases from gain of function mutations in SOD1  Rodents over-expressing SOD1 recapitulate ALS 2002: Minocycline reported by a number of groups to extend survival of SOD1 mice 2003: Randomized placebo controlled trial (412 patients treated for 9 months) 2007: Results of the trial are published - minocycline found to have a harmful effect on patients with ALS

“In the past five years we have screened more than 70 drugs in mice across 221 studies, using rigorous and appropriate statistical methodologies. While we were able to measure a significant difference in survival between males and females with great sensitivity, we observed no statistically significant positive (or negative) effects for any of the 70 compounds tested, including several previously reported as efficacious. “ Scott et al., Amyotroph Lateral Scler 2008; 9: 4-15 ALS Therapy Development Institute (ALS TDI) “…the majority of published effects are most likely measurements of noise in the distribution of survival means as opposed to actual drug effect.“

Scott et al., Amyotroph Lateral Scler 2008; 9: SOD1 G93A control mice The probability of seeing an effect by chance alone is significant even with 10 animals per group

Publication decisions and their possible effects on inferences drawn from tests of significance — or vice versa THEODORED. STERLING University of Cincinnati “There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs - an "error of the first kind“ - and is published.” Journal of the American Statistical Association, 1959; 54:30-34

Psychological Bulletin, 1979; 86: “For any given research area, one cannot tell how many studies have been conducted but never reported. The extreme view of the "file drawer problem" is that journals are filled with the 5% of the studies that show Type I errors, while the file drawers are filled with the 95% of the studies that show non-significant results.” The “File Drawer Problem” and Tolerance for Null Results ROBERT ROSENTHAL Harvard University

“We evaluated 340 articles included in prognostic marker meta-analyses (Database 1) and 1575 articles on cancer prognostic markers published in 2005 (Database 2). ……..Only five articles in Database 1 (1.5%) and 21 in Database 2 (1.3%) were fully ‘negative’ for all presented results in the abstract and without efforts to expand on non-significant trends or to defend the importance of the marker with other arguments.” European Journal of Cancer, 2007; 43:

 SOD1 G93A transgenic mice  Started at 10 weeks of age  i.p. 25 and 50 mg/kg/day  7 animals / group (females)  Not randomized  “The experimenter was blinded to the treatment protocol.”  SOD1 G93A transgenic mice  Started at 5 weeks of age  i.p. 10mg/kg/day  10 animals / group (sex?)  Not randomized  Not blinded The survival benefit of minocycline in the SOD1 G93A mouse model of ALS might be due to small sample size and/or Bias

What happens if we don’t improve research? We will:  Stifle advances in science and medicine  Abuse valuable resources  Fail current and future generations  Lose credibility / public support “Most people say that it is the intellect which makes a great scientist. They are wrong: it is character.” Albert Einstein

Lack of transparency in reporting Deficient experimental procedures Publication bias Human nature Poor reproducibility ××=× Education Attentiveness to bias; Good experimental design How do we improve research methods and reporting?

RFA-GM Training Modules to Enhance Data Reproducibility (R25) “This FOA will support creative educational activities with a primary focus on developing courses for skills development, specifically, training modules for graduate students, postdoctoral fellows, and beginning investigators designed to enhance data reproducibility.”

Google: NIGMS clearinghouse

Lack of transparency in reporting Deficient experimental procedures Publication bias Human nature Poor reproducibility ××=× Review transparency in reporting Education Attentiveness to bias; Good experimental design How do we improve research methods and reporting?

Evaluation of the strength of the data used in support of grant applications

June 20 – 21, 2012 Washington Plaza Hotel Washington DC R Editors Reviewers Investigators Funders

“To ease the interpretation and improve the reliability of published results we will more systematically ensure that key methodological details are reported, and we will give more space to methods sections. We will examine statistics more closely and encourage authors to be transparent, for example by including their raw data.”

1.Rigorous statistical analysis 2.Transparency in reporting  Journals should have no limit or generous limits on the length of methods sections  Journals should use a checklist during editorial processing to ensure the reporting of key methodological and analytical information to reviewers and readers. 3.Data and material sharing 4.Consideration of refutations 5.Consider establishing best practice guidelines

Lack of transparency in reporting Deficient experimental procedures Publication bias Human nature Poor reproducibility ××=× Review transparency in reporting Culture Focus on rigor not glitter Education Attentiveness to bias; Good experimental design How do we improve research methods and reporting?

New Biographical Sketch Format Required for NIH and AHRQ Grant Applications Submitted for Due Dates on or After May 25, 2015  The new format extends the page limit for the biosketch from four to five pages.  Allows researchers to describe up to five of their most significant contributions to science, along with the historical background that framed their research.  Investigators can outline the central findings of prior work and the influence of those findings on the investigator’s field. NOT-OD

“Experiments framed by hypotheses establish the idea of “positive” data versus “negative” data. Positive data are consistent with the hypothesis and negative data falsify the hypothesis. This creates a potential bias to amass positive data, owing to the desire to avoid falsification.”

Lack of transparency in reporting Deficient experimental procedures Publication bias Human nature Poor reproducibility ××=× Review transparency in reporting Culture Focus on rigor not glitter Better reproducibility ××= Education Attentiveness to bias; Good experimental design How do we improve research methods and reporting?

“It is not the slowness with which conclusions are arrived at that should give satisfaction to the moral sense, but the thoroughness, the completeness, the all-sidedness, the impartiality, of the investigation.” Thomas Chrowder Chamberlin 1897