1. Increasing openness, reproducibility, and prediction in social science research
My general substantive interest in the gap between values and practices. The work that I am discussing today is a practical application of this interest to the gap between scientific values and practices. In particular, how can I best advance knowledge and my career at the same time?
Challenges I face when working to advance scientific knowledge and my career at the same time. And, how my scientific practices can be adapted to meet my scientific values.
Brian Nosek
University of Virginia -- Center for Open Science

2. Norms Counternorms Communality Secrecy Open sharing Closed
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume

3. Norms Counternorms Communality Universalism Secrecy Particularlism
Open sharing
Universalism
Evaluate research on own merit
Secrecy
Closed
Particularlism
Evaluate research by reputation
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume

4. Norms Counternorms Communality Universalism Disinterestedness Secrecy
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume

5. Norms Counternorms Communality Universalism Disinterestedness
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against one’s prior work
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own theories, findings
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume

6. Norms Counternorms Communality Universalism Disinterestedness
Open sharing
Universalism
Evaluate research on own merit
Disinterestedness
Motivated by knowledge and discovery
Organized skepticism
Consider all new evidence, even against one’s prior work
Quality
Secrecy
Closed
Particularlism
Evaluate research by reputation
Self-interestedness
Treat science as a competition
Organized dogmatism
Invest career promoting one’s own theories, findings
Quantity
Communality – open sharing with colleagues; Secrecy
Universalism – research evaluated only on its merit; Particularism – research evaluated by reputation/past productivity
Disinterestedness – scientists motivated by knowledge and discovery, not by personal gain; self-interestedness – treat science as a competition with other scientists
Organized skepticism – consider all new evidence, theory, data, even if it contradicts one’s prior work/point-of-view; organized dogmatism – invest career in promoting one’s own most important findings, theories, innovations
Quality – seek quality contributions; Quantity – seek high volume

7. Anderson, Martinson, & DeVries, 2007

8. Challenges
Low power Flexibility in analysis Selective reporting Ignoring nulls Lack of replication
Examples from:
Button et al – Neuroscience
Ioannidis – why most results are false (Medicine)
GWAS
Biology
Two possibilities are that the percentage of positive results is inflated because negative results are much less likely to be published, and that we are pursuing our analysis freedoms to produce positive results that are not really there. These would lead to an inflation of false-positive results in the published literature.
Some evidence from bio-medical research suggests that this is occurring. Two different industrial laboratories attempted to replicate 40 or 50 basic science studies that showed positive evidence for markers for new cancer treatments or other issues in medicine. They did not select at random. Instead, they picked studies considered landmark findings. The success rates for replication were about 25% in one study and about 10% in the other. Further, some of the findings they could not replicate had spurred large literatures of hundreds of articles following up on the finding and its implications, but never having tested whether the evidence for the original finding was solid. This is a massive waste of resources.
Across the sciences, evidence like this has spurred lots of discussion and proposed actions to improve research efficiency and avoid the massive waste of resources linked to erroneous results getting in and staying in the literature, and about the culture of scientific practices that is rewarding publishing, perhaps at the expense of knowledge building. There have been a variety of suggestions for what to do. For example, the Nature article on the right suggests that publishing standards should be increased for basic science research.
[It is not in my interest to replicate – myself or others – to evaluate validity and improve precision in effect estimates (redundant). Replication is worth next to zero (Makel data on published replications; motivated to not call it replication; novelty is supreme – zero “error checking”; not in my interest to check my work, and not in your interest to check my work (let’s just each do our own thing and get rewarded for that)
Irreproducible results will get in and stay in the literature (examples from bio-med). Prinz and Begley articles (make sure to summarize accurately)
The Nature article by folks in bio-medicine is great. The solution they offer is a popular one in commentators from the other sciences -- raise publishing standards.
Sterling, 1959; Cohen, 1962; Lykken, 1968; Tukey, 1969; Greenwald, 1975; Meehl, 1978; Rosenthal, 1979

9. Figure credit: fivethirtyeight.com
Silberzahn et al., 2015

11. Median effect size (d) = .29 % p < .05 = 63%
Reported Tests (122)
Median p-value = .02
Median effect size (d) = .29
% p < .05 = 63%
Unreported Tests (147)
Median p-value = .35
Median effect size (d) = .13
% p < .05 = 23%
We find that about 40% of studies fail to fully report all experimental conditions and about 70% of studies do not report all outcome variables included in the questionnaire. Reported effect sizes are about twice as large as unreported effect sizes and are about 3 times more likely to be statistically
significant.
N = 32 studies in psychology
Unreported tests (147)
Median p-value = .35
Median d = .13
% significant = 23%
Reported tests (N = 122)
Median p = .02
Median d = .29
% sig p<.05 = 63%
Franco, Malhotra, & Simonovits, 2015, SPPS

12. Positive Result Rate dropped from 57% to 8% after preregistration required.

13. 97%
37% xx
Open Science Collaboration, 2015, Science

14. Dreber et al.,
2016, PNAS

15. Fig. 3. Probability of a hypothesis being true at three different stages of
testing: before the initial study (p0), after the initial study but before the
replication (p1), and after replication (p2). “Error bars” (or whiskers) represent
range, boxes are first to third quartiles, and thick lines are medians.
Initially, priors of the tested hypothesis are relatively low, with a median of
8.8% (range, 0.7–66%). A positive result in an initial publication then moves
the prior into a broad range of intermediate levels, with a median of 56%
(range, 10–97%). If replicated successfully, the probability moves further up,
with a median of 98% (range, 93.0–99.2%). If the replication fails, the
probability moves back to a range close to the initial prior, with a median of
6.3% (range, 0.01–80%).
Dreber et al.,
2016, PNAS

16. 98%
56%
Fig. 3. Probability of a hypothesis being true at three different stages of
testing: before the initial study (p0), after the initial study but before the
replication (p1), and after replication (p2). “Error bars” (or whiskers) represent
range, boxes are first to third quartiles, and thick lines are medians.
Initially, priors of the tested hypothesis are relatively low, with a median of
8.8% (range, 0.7–66%). A positive result in an initial publication then moves
the prior into a broad range of intermediate levels, with a median of 56%
(range, 10–97%). If replicated successfully, the probability moves further up,
with a median of 98% (range, 93.0–99.2%). If the replication fails, the
probability moves back to a range close to the initial prior, with a median of
6.3% (range, 0.01–80%).
8.8%
6.3%
Dreber et al.,
2016, PNAS

17. Testing and Evaluation Support
Preregistration: Protocols, Analysis Plans

18. PREREGISTRATION Context of Justification Confirmation Data independent
Hypothesis testing
Context of Discovery Exploration Data contingent Hypothesis generating
p-values interpretable
p-values NOT interpretable
PREREGISTRATION
Presenting exploratory as confirmatory increases publishability of results at the cost of credibility of results
Study 1
Study 1
Study 2

19. Preregistration Challenge http://cos.io/prereg

20. Testing and Evaluation Support
Preregistration: Protocols, Analysis Plans
Open Science Framework: Workflow and data management and preservation

22. OpenSesame

23. OpenSesame

24. OSF osf.io Application Framework journals registries preprint servers
Workflow
Authentication
Permissions
File Storage
File Rendering
Persistence
Meta-database Integrations
Search
SHARE Data
osf.io
journals
registries
preprint servers
grants management
OSF is actually an application framework
It is a public good and scholarly commons
It supports the interface you see if you google OSF
Blog engine, Slide sharing (osf.io/meetings)
Workflow integration
Authentication
File storage
File rendering
Database
Metadata/Annotations/Commenting
External service integrations
Search
SHARE Data
peer review services
curation, annotation

25. Testing and Evaluation Support
Preregistration: Protocols, Analysis Plans
Open Science Framework: Workflow and data management and preservation
Registered Reports: Journal management and recruiting

26. Registered Reports PEER REVIEW Design Collect & Analyze Report Publish
Review of intro and methods prior to data collection; published regardless of outcome
Beauty vs. accuracy of reporting
Publishing negative results
Conducting replications
Peer review focuses on quality of methods
Committee Chair: Chris Chambers

27. Registered Reports 39 journals so far
AIMS Neuroscience
Attention, Percept., & Psychophys
Cognition and Emotion
Cognitive Research
Comp. Results in Social Psychology
Cortex
Drug and Alcohol Dependence
eLife
Euro Journal of Neuroscience
Experimental Psychology
Human Movement Science
Int’l Journal of Psychophysiology
Journal of Accounting Research
Journal of Business and Psychology
Journal of Euro. Psych. Students
Journal of Expt’l Political Science
Journal of Personnel Psychology
Journal of Media Psychology
Leadership Quarterly
Nature Human Behaviour
Nicotine and Tobacco Research
NFS Journal
Nutrition and Food Science Journal
Perspectives on Psych. Science
Royal Society Open Science
Social Psychology
Stress and Health
Work, Aging, and Retirement
Review of intro and methods prior to data collection; published regardless of outcome
Beauty vs. accuracy of reporting
Publishing negative results
Conducting replications
Peer review focuses on quality of methods
Committee Chair: Chris Chambers

28. Big Picture: Phase 1, Cycle 1
Oct-Nov: Launch, information gathering, define testing and evaluation plan
Nov-Mar: OSF & preregistration training, preregister initial studies, Registered Reports process
Apr-Aug: Capture data and workflow, share across teams
Aug-Nov: Facilitate replications
Nov-Dec: Facilitate comparing predictions and original, replication outcomes

29. Email matt.spitzer@cos.io the journal titles most relevant to you
Right now
the journal titles most relevant to you

30. Next steps Share your proposals, study approach with matt@cos.io
Scheduling education and training for OSF and Preregistration
Developing journal partnerships for Registered Reports

31. Contact info
Brian Nosek,
Matt Spitzer,
NGS2 Poster shared:
These slides shared: