1. Stephanie Wykstra March 7-8, 2016 Research Transparency Workshop Maanzoni Lodge, Athi River, Kenya Research reliability: The problems and the way forward

2. Part I: Problems

3. Kick-off discussion Take a few minutes to write down your answer to these two questions. 1) When you read or hear about a new piece of research (i.e. in a publication or a talk), what are things that would make you trust the result? What would make you skeptical? 2) Are there any particular cases where you tried to delve into the research result to investigate it further? What did you do, and how did it go?

4. Reliable research Research results that we can rely on are incredibly important. Policies, programs and products (such as pharmaceutical drugs) based on research affect many millions of lives. … So the question is: can we confidently rely on research results? What problems undermine our ability to rely on scientific research? And how can we do better?

5. Research problems: Publication bias P-hacking Non-disclosure Selective reporting Failure to replicate Lack of transparency

6. Publication Bias “File drawer problem”

7. Publication Bias  Statistically significant results more likely to be published, while null results are buried in the file drawer.

8. Very few studies report null results Ed Yong, Nature 2012Nature 2012

9. Publication Bias Example: Franco et al. (Science 2014) found that of a group of NSF-funded studies run through TESS (time-sharing experiments in the social sciences), 60% of those with positive, statistically significant results had been published versus only 20% of those finding null results.Science 2014

10. In social sciences… Franco et al. (Science 2014)Science 2014

11. How would you respond to this person? “I don’t see the problem if we don’t see the null results. We’re interested in finding out what works, and reading about studies where the researchers didn’t find anything significant to report won’t help us much! …. So what’s the problem?” Pop quiz

12. Also called “data fishing,” “data mining.” “If you torture your data long enough, they will confess to anything.” -Ronald Coase P-Hacking

13. Researchers test hypotheses i.e. look for relationships among variables (e.g. schooling, test scores).  In particular, a result which shows a result which is statistically significant at p<.05 is often considered noteworthy and thus more publishable. This leads to pressure to tinker with specifications (how variables are coded, which outliers are excluded, which subgroups are considered, etc) to find p<.05.

14. “Why Most Research Findings Are False” Ioannidis (PLOS 2005)PLOS 2005 “The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.” P-Hacking

15. Turner et al. [2008] In economics… Brodeur et al (AEJ 2016). Data: 50,000 tests published in AER, JPE, QJE (2005-2011)AEJ 2016

16. In sociology… Gerber and Malhotra 2008 doi: 10.1177/0049124108318973

17. In political science… Gerber and Malhotra 2008 doi: 10.1561/100.00008024

18. Selective Reporting Cherry-picking results for reporting

19. Selective Reporting XKCD http://xkcd.com/882/

20. Selective Reporting Malhotra, Franco, Simonovits (2015) find that of the studies run through TESS, roughly 60% of papers report fewer outcome variables than are listed in the questionnaire. If many relevant outcomes aren’t mentioned in the paper (and aren’t listed elsewhere), how can we be confident that the reported results weren’t just noise, rather than true effects?

21. Replications “Replication” is often used to mean different things. Here are three different activities which replication refers to: (1) Verification and re-analysis: Checking that original data/code can produce published results, as well as going further to check the robustness of the results. (2) Reproduction: Testing whether the results hold up when the study conducted in a very similar way. (3) Extension: Investigating whether the results hold up when the study is conducted in another place, under different conditions, etc., to test external validity.

22. Replication Terminology from Michael Clemens 2014Michael Clemens 2014

23. Well-known replications: Reinhart-Rogoff (2013)  Spreadsheet errors found by grad student (Herndon et al 2013).Herndon et al 2013 Deworming debate (2015)  Miguel/Kremer (2004) and Aiken/Davey et al. (2015) Miguel/KremerAiken/Davey et al.  Big debate within development econ/epidemiology Reproducibility project in psychology (2015) Reproducibility project  ~40% of studies successfully reproduced.  But big debate this past week! What does it mean for a replication to “fail”? Begley et al. (2012): Begley et al  Attempt to reproduce “landmark” pre-clinical cancer lab studies at Amgen (6 out of 53 studies reproduced).

24. Replication incentives Journals are often reluctant to publish “unoriginal” work, and replications are often considered “unoriginal.” Yet simply sharing results of replications on a website (e.g. Political Science Replication Initiative; Psych file drawer) is not ideal either: some argue strongly that replications should be peer-reviewed.

25. Transparency Peng 2011, Science 334(6060) pp. 1226-1227

26. Transparency Sharing data, code and surveys (along with clear documentation) can allow others to check one’s work.

27. Transparency Yet relatively few researchers sharing data:  Alsheichk-Ali et al. 2011: Review of 10 first research papers of 2009 published in top 50 journals by impact factor. Of the 500 papers, 351 were subject to a data availability policy of some kind: 59% did not adhere to the policy, most commonly by not publicly depositing the data (73%). Overall, only 47 papers (9%) deposited the full primary raw data online. Alsheichk-Ali et al. 2011 Research funders (e.g. government funders such as NSF, NIH, ERSC in the UK, private foundations) are adopting data- sharing policies at greater rates. Yet they rarely enforce their policies.

28. Transparency And relatively few journals have data-sharing policies: In 2013, only 18 out of 120 political science journals inspected had data-sharing policies (Gherghinaa and Katsanidoua 2013) and another review found that only 29 out of 141 journals reviewed in economics had policies (Vlaeminck 2013).Gherghinaa and Katsanidoua 2013Vlaeminck 2013 Even though having a mandatory policy makes it much more likely that researchers will share data:  The rate of data archiving compliance for surveyed journals with the strictest policies, which required data archiving along with data accessibility statements in manuscripts, was nearly 1000x higher than having no policy. (Vines 2014)Vines 2014

29. Discussion Discuss in groups of 3-4: What changes in how research is practiced and shared do you think would improve the reliability of the research the most? Why?

30. Part II: Solutions

31. As we have discussed…big problems include:  Publication bias  Data-mining  Selective reporting  Lack of transparency and inability to replicate Meta-research

32. Research transparency! 1. Study registration and pre-analysis plans 2. “Results-neutral publishing” 3. Replications 4. Publish / share all study results 5. Data-sharing: sharing data, code, surveys, readme files How do we improve?

33. Creating a public record of a study and basic information about the study (e.g., intervention, outcomes, location, dates). All IPA required and J-PAL studies strongly encouraged to pre-register studies on the AEA registry (www.socialscienceregistry.org). We recommend registering prior beginning the intervention. Registering studies

34. A website for registering RCTs in the social sciences, established by the American Economics Association Established 2013 ~580 trials registered. What is the AEA registry?

36. Other registries:

37. Reward for registration

38. Combats publication bias by providing public record of the study. Pre-registration is a long-standing requirement in the medical community (e.g., for FDA approval, for publication in medical journals) and is increasingly a focus in the social sciences as well. Clinicaltrials.gov (185K+ studies registered). Why register a trial?

39. How and when to register? How? Register by making a user account on socialscienceregistry.org. 18 required fields: basic information such as trial title, country, intervention start/end dates, outcomes, etc. Project staff (e.g. research managers) may register a trial for Pis, but PIs must log in to approve and submit the registration. More detailed “how to” guide and video in Box [link] / Sharepoint.link

40. Pre-analysis plans Pre-analysis plans are more detailed write-ups about the study hypotheses, outcomes, and planned analysis. The goal is to combat data-mining by tying the hands of the researcher. AEA registry does NOT require pre-analysis plans (about 1/6 of registered studies have them – you can search by “pre- analysis plan” field if interested).

41. Pre-analysis plans Why have a pre-analysis plan? We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000. (Kaplan and Irvin, 2015, PLOS).Kaplan and Irvin

42. Pre-analysis plans Kaplan and IrvinKaplan and Irvin, 2015, PLOS

43. Share all results (including null results!) Publication bias leads to a skew towards positive (exciting) results. Problem: how do we disseminate null results, if journals tend to accept positive results at greater rates? One solution: reporting all results in a public registry. For example: the FDA in the US requires that all results be reported on clinicaltrials.gov within one year after trial completion. For social sciences studies, the results could be reported in the place they are registered (e.g. AEA registry). Alltrials is a group that campaigns to push for all clinical trials to report results.

44. Results-neutral publishing Submitting a protocol with details on methodology of study and planned analysis, before carrying out the study. The article is then accepted based on the design, before any data are collected. “Because the study is accepted in advance, the incentives for authors change from producing the most beautiful story to producing the most accurate one.” –Chris Chambers, editor of Cortex (OSF citation) citation

45. Results-neutral publishing 20 journals have adopted the “registered reports” model so far (list). Examples:list AIMS Neuroscience Attention, Perception & Psychophysics Comparative Political Studies Comprehensive Results in Social Psychology Cortex Drug and Alcohol Dependence eLife Experimental Psychology Frontiers in Cognition Journal of Business and Psychology Nutrition and Food Science Journal Perspectives on Psychological Science Social Psychology Working, Aging, and Retirement

46. Replications As valuable as they are, replications are relatively rare. What can the research community do to increase replications being done and shared?

47. Replication initiatives: 3ie’s replication program Reproducibility projects in psychology and cancer biology Many labs project in psychology Replication wiki in economics

48. What? Making data, code, and other materials publicly available. "An article...in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete set of instructions [and data] which generated the figures.” - Dr. David Donoho, Stanford stats professor Data-sharing

49. Facilitates replication of published results Re-use of data for further studies and meta-analysis Promote better quality data/code/metadata Data-sharing: why?

50. Funders: Holdren memo, February 2013 Foundations – Gates, Wellcome Trust (and 15 other funders who signed joint statement), many others Journals and associations Top economics journals e.g. AER, Econometrica, AEJ American Political Science Association - new ethics guidelines for data-sharing Open science groups advocating for open data Berkeley Initiative for Transparency in the Social Sciences Center for Open Science Meta-research institute at Stanford (Ioannidis) YODA at Yale (clinical trials data-sharing portal) The data-sharing movement

51. IPA has adopted new data publication guidelines and policy: sharing data by 3 years following endline data collection. Data repository created for data from RCTs (including IPA/J-PAL and other collaborating organizations). Research transparency at IPA

52. Data from RCTs in the social sciences

54. IPA data publication guidelines [link]link IPA guidelines

55. 1. Datasets Recommended: cleaned, study dataset (PII removed!) Minimally: “publication” dataset – code/data underlying publication. Ideal: start to finish reproducibility (more on this in a minute!) 2. Readme files explaining relation between data and code, as well as any further data documentation. 3. Surveys 4. Study-level metadata What to share?

56. Data repositories: Dataverse (used by IPA and J-PAL) Can also create a personal dataverse (free!) such as Ted Miguel’s Dataverse. Figshare Dryad ICPSR (not free to deposit data, but curation services are offered). Many others for specific types of data, see Re3data.org Re3data.org Where to share?

57. 1. Checking usability of data e.g., that there are variable labels/value codes 2. Ensuring no PII is publicly released: Checking that all direct identifies (names, IDs, etc) are removed. Indirect identifiers such as geospatial data removed. Thinking carefully about whether variables can be recombined to identify participants. 3. Replicating the study, in the sense of checking that data/code produce the published results 4. Checking that ReadMe files provide good instructions for replication 5. Ensuring variable/file/study-level metadata is shared. Steps before sharing data/code:

58. The ideal: sharing “start to finish” data/code to permit replication from initial raw dataset to final tables. Includes all code (variable construction and cleaning as well). Key point: sharing isn’t all or nothing: sharing some data/code e.g. underlying the published results … is better than sharing nothing! What is “fully reproducible research”?

59. Preparing data/code early on is crucial, since there are serious limitations in what can be shared and understood later on, if good practices aren’t followed. Getting data/code into shape early on

60. e.g., Variable naming and labeling Commenting code Master do files Headers/footers ReadMe files …More detail in IPA’s best practice manual for managing data and code.best practice manual Getting data/code into shape

61. Version control for code using git (session later today) Dynamic documents using Markdoc (Stata) or Rmarkdown (R) which generate tables in papers directly from the code (session tomorrow) Open science framework for collaborative workflow (session tomorrow) Tools for reproducible research

62. Badges to reward open practices Incentives to share data? https://osf.io/tvyxz

63. Data citation and shifting tenure/promotion committees to rewarding transparency, rather than just publication. Incentives to share data?

64. Transparency and openness promotion (TOP) guidelines (Science, 2015)Science Incentives to share data?

65. We have talked about ways of improving the reliability of research (sharing data/code, replicating work, pre- registering studies and reporting all results): What do you think the main obstacles are, which would prevent researchers from doing these things? What are the most promising ways to improve the reliability of research, in your view? Which are the least promising? Discussion