1. Improving Openness and Reproducibility of Scientific Research

2. Norms Counternorms Communality Universalism Disinterestedness
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against one’s prior work
Quality
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own theories, findings
Quantity
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume
Merton, 1942

3. Norms Counternorms Communality Universalism Disinterestedness
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against one’s prior work
Quality
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own theories, findings
Quantity
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume
Merton, 1942

4. Norms Counternorms Communality Universalism Disinterestedness
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against one’s prior work
Quality
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own theories, findings
Quantity
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume
Merton, 1942

5. Norms Counternorms Communality Universalism Disinterestedness
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against one’s prior work
Quality
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own theories, findings
Quantity
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume
Merton, 1942

6. Norms Counternorms Communality Universalism Disinterestedness
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against one’s prior work
Quality
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own theories, findings
Quantity
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume
Merton, 1942

7. Anderson, Martinson, & DeVries, 2007

8. Anderson, Martinson, & DeVries, 2007

9. Anderson, Martinson, & DeVries, 2007

10. Incentives for individual success are focused on getting it published, not getting it right
Nosek, Spies, & Motyl, 2012

11. Flexibility in analysis
Problems
Flexibility in analysis
Selective reporting
Ignoring nulls
Lack of replication
Examples from:
Button et al – Neuroscience
Ioannidis – why most results are false (Medicine)
GWAS
Biology
Two possibilities are that the percentage of positive results is inflated because negative results are much less likely to be published, and that we are pursuing our analysis freedoms to produce positive results that are not really there. These would lead to an inflation of false-positive results in the published literature.
Some evidence from bio-medical research suggests that this is occurring. Two different industrial laboratories attempted to replicate 40 or 50 basic science studies that showed positive evidence for markers for new cancer treatments or other issues in medicine. They did not select at random. Instead, they picked studies considered landmark findings. The success rates for replication were about 25% in one study and about 10% in the other. Further, some of the findings they could not replicate had spurred large literatures of hundreds of articles following up on the finding and its implications, but never having tested whether the evidence for the original finding was solid. This is a massive waste of resources.
Across the sciences, evidence like this has spurred lots of discussion and proposed actions to improve research efficiency and avoid the massive waste of resources linked to erroneous results getting in and staying in the literature, and about the culture of scientific practices that is rewarding publishing, perhaps at the expense of knowledge building. There have been a variety of suggestions for what to do. For example, the Nature article on the right suggests that publishing standards should be increased for basic science research.
[It is not in my interest to replicate – myself or others – to evaluate validity and improve precision in effect estimates (redundant). Replication is worth next to zero (Makel data on published replications; motivated to not call it replication; novelty is supreme – zero “error checking”; not in my interest to check my work, and not in your interest to check my work (let’s just each do our own thing and get rewarded for that)
Irreproducible results will get in and stay in the literature (examples from bio-med). Prinz and Begley articles (make sure to summarize accurately)
The Nature article by folks in bio-medicine is great. The solution they offer is a popular one in commentators from the other sciences -- raise publishing standards.
Sterling, 1959; Cohen, 1962; Lykken, 1968; Tukey, 1969; Greenwald, 1975; Meehl, 1978; Rosenthal, 1979

12. Figure by FiveThirtyEight.com Silberzahn et al., 2015

13. A Garden of Forking Paths
“Does X affect Y?”
Exclude outliers?
Control for year?
Median or mean?
A series of perfectly reasonable decisions coupled with motivated reasoning can quickly lead us to a subset of statistically significant results. In effect, our hypothesis changed without us even realizing it. Our confirmatory, hypothesis testing became exploratory, hypothesis generating without our permission!
Jorge Luis Borges; Gelman and Loken

14. Franco, Malhotra, & Simonovits, 2015, SPPS
We find that about 40% of studies fail to fully report all experimental conditions and about 70% of studies do not report all outcome variables included in the questionnaire. Reported effect sizes are about twice as large as unreported effect sizes and are about 3 times more likely to be statistically
significant.
N = 32 studies in psychology
Unreported tests (147)
Median p-value = .35
Median d = .13
% significant = 23%
Reported tests (N = 122)
Median p = .02
Median d = .29
% sig p<.05 = 63%
Franco, Malhotra, & Simonovits, 2015, SPPS

15. Median effect size (d) = .29 % p < .05 = 63%
Reported Tests (122)
Median p-value = .02
Median effect size (d) = .29
% p < .05 = 63%
We find that about 40% of studies fail to fully report all experimental conditions and about 70% of studies do not report all outcome variables included in the questionnaire. Reported effect sizes are about twice as large as unreported effect sizes and are about 3 times more likely to be statistically
significant.
N = 32 studies in psychology
Unreported tests (147)
Median p-value = .35
Median d = .13
% significant = 23%
Reported tests (N = 122)
Median p = .02
Median d = .29
% sig p<.05 = 63%
Franco, Malhotra, & Simonovits, 2015, SPPS

16. Median effect size (d) = .29 % p < .05 = 63%
Reported Tests (122)
Median p-value = .02
Median effect size (d) = .29
% p < .05 = 63%
Unreported Tests (147)
Median p-value = .35
Median effect size (d) = .13
% p < .05 = 23%
We find that about 40% of studies fail to fully report all experimental conditions and about 70% of studies do not report all outcome variables included in the questionnaire. Reported effect sizes are about twice as large as unreported effect sizes and are about 3 times more likely to be statistically
significant.
N = 32 studies in psychology
Unreported tests (147)
Median p-value = .35
Median d = .13
% significant = 23%
Reported tests (N = 122)
Median p = .02
Median d = .29
% sig p<.05 = 63%
Franco, Malhotra, & Simonovits, 2015, SPPS

17. Estimating Reproducibility
Increasing Depth
Increasing Breadth

21. Reproducibility in other fields
Developmental Psychology
Michael Frank
Ecology
Emilio Bruna
Health Sciences
Computer Science
Leslie McIntosh
Cynthia Hudson-Vitale
Christian Collberg Todd Proebsting

22. Solution? Appeal to intentions, values, and goals.
“Hey You! Behave by your values! Be objective!”

23. Incentives for individual success are focused on getting it published, not getting it right
Nosek, Spies, & Motyl, 2012

24. Rewards What is published? What is not? Replications Novel results
Negative results
Mixed evidence
Novel results
Positive results
Clean results

27. Evidence to encourage change Technology to enable change
Training to enact change
Incentives to embrace change
Improving scientific ecosystem

28. Infrastructure
Metascience
Community

29. Infrastructure

30. Technology to enable change
Supporting these behavioral changes requires improving the full scientific ecosystem.
At a conference like IDCC, there are many people in the room contributing important parts to this ecosystem. I hope you leave this talk seeing the potential for how we might be able to work together on connecting tools to provide for better transparency and reproducibility in the workflow.

33. Collaboration Documentation Archiving
Content management and collaboration system
Free service
Connect, curate, search all aspects of the research project
We don’t want to repeat ourselves
Dataverse S3
Figshare
Dropbox
Service vs Application interface
the service build new applicatoins
this is right in line with SHARE-NS - openning, unlcoking this data
allows for innovation
From day one, we’ve been very excited about the SHARE partnerhship:
community, expertise bringing to the table
as well as
shared interests
mission
Technical perspective, this project fit very much in line with what are building and more importantly how we are building it

34. Content management and collaboration system
Free service
Connect, curate, search all aspects of the research project
We don’t want to repeat ourselves
Dataverse S3
Figshare
Dropbox
Service vs Application interface
the service build new applicatoins
this is right in line with SHARE-NS - openning, unlcoking this data
allows for innovation
From day one, we’ve been very excited about the SHARE partnerhship:
community, expertise bringing to the table
as well as
shared interests
mission
Technical perspective, this project fit very much in line with what are building and more importantly how we are building it

35. Version Control

37. Merges Public-Private Workflows

39. Incentives for Openness File downloads
We can do small things like offer rich analytics to incentivize more open practices. Public projects gain immediate access to analytics showing visits over time, sources of traffic, geographic location of traffic, time of traffic, and download counts for files. This is a much faster reward for one’s effort than waiting months or longer until something is published and then even longer until it is cited.

40. File downloads
We can do small things like offer rich analytics to incentivize more open practices. Public projects gain immediate access to analytics showing visits over time, sources of traffic, geographic location of traffic, time of traffic, and download counts for files. This is a much faster reward for one’s effort than waiting months or longer until something is published and then even longer until it is cited.
Forks

41. Persistent
Citable
Identifiers

43. Registration

45. Connecting the workflow is critical to enabling change

46. Search and discover Publish report Write report Develop idea
Interpret findings
Design study
Analyze Data
Acquire materials
There’s more to it than sharing of discrete objects. Think about using this as an opportunity to increase transparency by capturing the entire workflow, and to do so while connecting tools and services that make up the parts of the workflow, not requiring people to change all of their practices at once, and providing immediate efficiencies and value to the researcher AS they comply with requirements.
Easy, right? Obviously not.
Store data
Collect data

47. Search and discover Publish report Write report Develop idea
Interpret findings
Design study
Analyze Data
Acquire materials
There’s more to it than sharing of discrete objects. Think about using this as an opportunity to increase transparency by capturing the entire workflow, and to do so while connecting tools and services that make up the parts of the workflow, not requiring people to change all of their practices at once, and providing immediate efficiencies and value to the researcher AS they comply with requirements.
Easy, right? Obviously not.
Store data
Collect data
OpenSesame

48. Community

52. Training to enact change
Once infrastructure is in place, we need to show researchers how to use it to improve their practices.

53. Free training on how to make research more reproducible
Partner with others on training --- librarians are great partners in this ---- to teach researchers skills in how to deal with basic data management and how to improve their research workflows for personal and sharing purposes.
Software Carpentry and Data Carpentry are other great examples of efforts in this area, and partnerships with those in libraries --- we’ve done some work with them and are exploring ways to do more.
Free training on how to make research more reproducible

55. Incentives to embrace change
Supporting these behavioral changes requires improving the full scientific ecosystem.
At a conference like IDCC, there are many people in the room contributing important parts to this ecosystem. I hope you leave this talk seeing the potential for how we might be able to work together on connecting tools to provide for better transparency and reproducibility in the workflow.

56. Transparency & Openness Promotion Guidelines
Agnostic to discipline
Low barrier to entry
Modular

57. Transparency & Openness Promotion Guidelines
Eight Standards
Data Citation
Design transparency
Research materials
Data
Analytical methods
Preregistered studies
Preregistered analysis plans
Registered Reports

58. Transparency & Openness Promotion Guidelines
Eight Standards
Data Citation
Design transparency
Research materials
Data
Analytical methods
Preregistered studies
Preregistered analysis plans
Registered Reports
Three Tiers
Disclose Require Verify

59. Transparency & Openness Promotion Guidelines
Eight Standards
Data Citation
Design transparency
Research materials
Data
Analytical methods
Preregistered studies
Preregistered analysis plans
Registered Reports
Three Tiers
Disclose Require Verify

60. Transparency & Openness Promotion Guidelines
Eight Standards
Data Citation
Design transparency
Research materials
Data
Analytical methods
Preregistered studies
Preregistered analysis plans
Registered Reports
Three Tiers
Disclose Require Verify

61. Transparency & Openness Promotion Guidelines
Signatories
539 Journal signatories
59 Organizational signatories
Learn more at

62. Signals: Making Behaviors Visible Promotes Adoption
Badges
Open Data
Open Materials
Preregistration
Psychological Science (Jan 2014)

64. 40%
30%
% Articles reporting that data was available
20%
10% 0%

65. 100% 75% 50% 25% 0% % of Articles reporting that data was available
On the y axis, we have % of articles reporting data available in an independent, open access location
On the x axis, we have five categories: reportedly available, accessible, correct, usable, and complete data
25% 0%
Reportedly Available Accessible Correct Data Usable Data Complete Data

66. 100% 75% 50% 25% 0% % of Articles reporting that data was available
In an ideal world, we’d see straight lines for articles published in all journals, and in psychological science before and after badges
This would mean that all articles that reported available data had data that was accessible, correct, usable, and complete
25% 0%
Reportedly Available Accessible Correct Data Usable Data Complete Data

67. 100% 75% 50% 25% 0% % of Articles reporting that data was available
In Psychological Science prior to badges and in other comparison conditions, only 39% or less (20%, 16%) of articles with reportedly available data in the comparison conditions had data that was accessible, correct, usable, and complete
25% 0%
Reportedly Available Accessible Correct Data Usable Data Complete Data

68. 100% 75% 50% 25% 0% % of Articles reporting that data was available
Of articles in Psychological Science that earned badges, over 75% of articles with data reportedly available had data that was accessible, correct, usable, and complete
While not perfect, reportedly available data were more likely to be persistent when badges were earned -- perhaps due to the accountability of receiving a badge on the publication.
25% 0%
Reportedly Available Accessible Correct Data Usable Data Complete Data

69. The $1,000,000 Preregistration Challenge
Another incentive for researchers to try out preregistration.

70. Exploratory research:
Finds unexpected trends
Pushes knowledge into new areas
Results in a testable hypothesis

71. Confirmatory research:
Puts a hypothesis to the test
Does not allow data to influence the hypothesis
Results are held to the highest standard of rigor

72. https://cos.io/prereg

74. Data collection methods
Research questions
Data collection methods
Variables
Statistical tests
Outliers

75. Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Positive result rate dropped from 57% to 8% after preregistration became required for clinical trials.
Kaplan and Irvin, 2015

76. Registered Reports Design Collect & Analyze Report Publish PEER REVIEW
Review of intro and methods prior to data collection; published regardless of outcome
Beauty vs. accuracy of reporting
Publishing negative results
Conducting replications
Peer review focuses on quality of methods

77. Registered Reports Design Collect & Analyze Report Publish PEER REVIEW
Review of intro and methods prior to data collection; published regardless of outcome
Beauty vs. accuracy of reporting
Publishing negative results
Conducting replications
Peer review focuses on quality of methods

78. Who Publishes Registered Reports?
So.. who publishes these things?
Here’s a partial (and growing!) list. You can view the complete list on the Registered Reports project page on the OSF. There’s even a table comparing features of RRs across journals.
(just to name a few)
See the full list and compare features: osf.io/8mpji

79. http://osf.io/8mpji eLife Chris chambers slide/ committee
16 journals incl. eLife
Special issue

80. https://osf.io/x5w7h Find this presentation at
(contact information)