1. Challenges for Journals: Encouraging Sound Science
Barbara A. Spellman
University of Virginia School of Law
There is a war between the ones who say there is a war and the ones who say there isn’t…

2. 2009: Would you like to be Editor of Perspectives on Psychological Science? Sure !
Jan 2010 – Letter from Incoming Editor 1. Thanks to previous editor New types of articles...

3. 2011 (Jan OL) – Bem, “Feeling the Future” 2011 (Oct Interim Rpt
2011 (Jan OL) – Bem, “Feeling the Future” 2011 (Oct Interim Rpt.) -- Stapel Fraud 2011 (Oct OL) – Simmons et al., “False-positive Psychology” (“QRPs”)

4. RETRACTION Failure to Replicate
Underpowered Studies
Failure to Replicate
RETRACTION
Fraud
Failure to Replicate
Failure to Replicate
File Drawer Problem
Failure to Replicate

5. Take your shot...

6. November 2012 Hal Pashler & EJ Wagenmakers & Me
Special Section on Replicability in Psychological Science
Special Section on Research Practices
170 pages
120 authors
Invited, collected, un-rejected, requested reply, bid for (at auction)

7. November 2012 Hal Pashler & EJ Wagenmakers & Me
Problems
Solutions
Is there a replication crisis ?
(Non-)Value of conceptual replications
Undead theories
Aesthetic standards
Too many successful “predictions”
It is not self-correcting
Teaching Replications
Rewarding Replication
Restructuring Incentives
Open to Change / Wary of Rules
Outline of the RP:P
What Journals Can Do

8. How...
...can journals help document the problems? Raise awareness of the problems?
...did journals contribute to the problems?
...can journals help to reduce the problems in the future?
...can journals help fix the problems they created in the past (i.e., fix the record)?

9. Why are there failures to replicate?
Fault of Original Researcher
Fault of Neither, One, or Both
Fault of Replicator
Fraud
QRPs (“Questionable Research Practices”)
Chance
Bad Copy (methods / data / analysis description)
Things have changed
QRPs
Bad intentions
Incompetence

10. The “Ideal”: Hypothetico-Deductive Model
Generate and specify hypotheses
Design study
Collect data
Analyse data & test hypotheses
Interpret data
Publish or conduct next experiment

11. External Influences Provide Bad Incentives
GET Job, Tenure, Promotion, Fame, Students…
External Influences Provide Bad Incentives
GET FUNDED
Generate and specify hypotheses
Design study
Collect data
Analyse data & test hypotheses
Interpret data
Publish or conduct next experiment
GET PUBLISHED

12. External Influences Provide Bad Incentives
GET Job, Tenure, Promotion, Fame, Students…
External Influences Provide Bad Incentives
GET FUNDED
Generate and specify hypotheses
Design study
Collect data
Analyse data & test hypotheses
Interpret data
Publish or conduct next experiment
GET PUBLISHED

13. What do Journals Want? To make money ?
Get people to buy / read / cite journal.
Important papers.
Novel papers.
Clear papers.
Papers confirm some new hypothesis.
Don’t spend too much.
Keep articles short.
Don’t spend lots on reviewing, fact checking, etc.

14. TO GET PUBLISHED What do Journals Want? Scientists To make money ?
Get people to buy / read / cite journal.
Important papers.
Novel papers.
Clear papers.
Papers confirm some new hypothesis.
Don’t spend too much.
Keep articles short.
Don’t spend lots on reviewing, fact checking, etc.

15. The Norm: Research Incentives and The Garden of Forking Paths
Novelty bias: lack of replication as a norm means that only 1 in 1000 papers are replicated
Generate and specify hypotheses
Design study
Collect data
Analyse data & test hypotheses
Interpret data
Publish or conduct next experiment
Publication bias – 92% positive results
Data, materials, method not shared (<30% data shared)
Peer review is secret
Changing the hypotheses, “HARKing” to fit the data or analyses
Work through these one-by-one
Can we put together an example research paradigm?
“Cherry picking” measures or constructs that support hypotheses
Mining data for statistically significant associations
Low statistical power, poor chance to detect effects even if they exist
Lack of pre-registration
“Cherry picking” data that supports hypotheses
Experimenter influences on data coding and analysis

16. Requiring “Novelty” distorts the literature

17. Requiring “Novelty” distorts the literature

18. How can I get beautiful results to support my novel hypothesis?
Improve your own luck !
Take risks with small samples.
Try a lot of things; pick the good cherries.

19. How can I get beautiful results to support my novel hypothesis?
Improve your own luck !
Take risks with small samples.
Try a lot of things; pick the good cherries.
Change your hypothesis to fit your data !
HARKing: Hypothesising after the results are known
Coined by Norbert Kerr (1998) to describe the practice of: “presenting post hoc hypotheses in a research report as if they were, in fact, a priori hypotheses”

20. HARKing and aesthetic standards
Generate and specify hypotheses
Design study
Collect data
Analyse data & test hypotheses
Interpret data
Publish or conduct next experiment
Changing the hypotheses, “HARKing” to fit the data or analyses
Work through these one-by-one
Can we put together an example research paradigm?

21. Keeping Methods Short: Hurts Ability to Evaluate and Replicate

22. Not Requiring Data Sharing or Material/Method Sharing or Code Sharing: Hurts Ability to Evaluate (+ can’t reproduce)

24. Change/Align Incentives
What is good for individual scientist’s career should be what is good for science itself.

25. How... ...did journals contribute to the problems?
...can journals help document the problems? Raise awareness of the problems?
...can journals help to reduce the problems in the future?
...can journals help fix the problems they created in the past (i.e., fix the record)?
Use: Incentives, Technology & Determination

26. I’d like to personally invite you to an important workshop titled, “Increasing Scientific Transparency and Reproducibility in the Social and Behavioral Sciences” being held on November 3-4, 2014 at the Center for Open Science (COS) in Charlottesville, VA. The workshop is funded by the Laura and John Arnold Foundation and co-hosted by the journal Science, the Berkeley Initiative for Transparency in the Social Sciences (BITSS), and COS.

27. Transparency and Openness Promotion (TOP) Guidelines
Eight policy guidelines for increasing the transparency and reproducibility of the published research.
Agnostic to discipline
Steps (Nothing, Disclose, Require, Verify) to lower barriers
Modular
See cos.io/top for more detailed language

28. Materials transparency Data transparency Code transparency
Level 1 Level 2 Level 3
Eight Standards
Data citation
Materials transparency
Data transparency
Code transparency
Design transparency
Study Preregistration
Analysis Preregistration
Replication
See cos.io/top for more detailed language

29. Materials transparency Data transparency Code transparency
Level 1 Level 2 Level 3
Eight Standards
Data citation
Materials transparency
Data transparency
Code transparency
Design transparency
Study Preregistration
Analysis Preregistration
Replication
What does it mean?
What does it solve?
How do we get authors to do it?
See cos.io/top for more detailed language

30. Materials transparency Data transparency Code transparency
Level 1 Level 2 Level 3
Eight Standards
Data citation
Materials transparency
Data transparency
Code transparency
Design transparency
Study Preregistration
Analysis Preregistration
Replication
Ask authors who submit to answer 2 questions:
Are the data/code/materials available in a public repository? Yes/No
If Yes, where: URL: ________
Make answers available in article metadata, or simply in footnotes.
See cos.io/top for more detailed language

31. Journal Carrots: Badges for Open Science
Provide more “room” for complete methods, materials, data presentations. (New Technology!) Award badges. Open Data, Materials, Analyses

33. Data, Analytic Methods (Code), and Research Materials Transparency
Level 1: Authors must disclose action

34. Data, Analytic Methods (Code), and Research Materials Transparency
Level 2: Authors must share (exceptions permitted)

35. Data, Analytic Methods (Code), and Research Materials Transparency
Level 3: Journal or third party will verify that the data can be used to reproduce the findings presented in a paper.

36. Design and Analysis Transparency
Society or journal defines the relevant reporting standards that are appropriate for their discipline.

37. Materials transparency Data transparency Code transparency
Level 1 Level 2 Level 3
Eight Standards
Data citation
Materials transparency
Data transparency
Code transparency
Design transparency
Study Preregistration
Analysis Preregistration
Replication
What does it mean?
What does it solve?
How do we get authors to do it?
See cos.io/top for more detailed language

38. Materials transparency Data transparency Code transparency
Level 1 Level 2 Level 3
Eight Standards
Data citation
Materials transparency
Data transparency
Code transparency
Design transparency
Study Preregistration
Analysis Preregistration
Replication
See cos.io/top for more detailed language

39. Preregistration
A preregistration is a time-stamped, read-only version of your research plan created before the study.
It increases credibility by specifying in advance how data will be analyzed, and makes the distinction between confirmatory and exploratory work more clear.

40. Alison ledgerwood

41. Confirmatory versus exploratory analysis
“In statistics, hypotheses suggested by a given dataset, when tested with the same dataset that suggested them, are likely to be accepted even when they are not true. This is because circular reasoning (double dipping) would be involved: something seems true in the limited data set, therefore we hypothesize that it is true in general, therefore we (wrongly) test it on the same limited data set, which seems to confirm that it is true. Generating hypotheses based on data already observed, in the absence of testing them on new data, is referred to as post hoc theorizing (from Latin post hoc, "after this").
The correct procedure is to test any hypothesis on a data set that was not used to generate the hypothesis.”

42. Confirmatory versus exploratory analysis
Preregistration
Context of confirmation
Traditional hypothesis testing
Results held to the highest standards of rigor
Goal is to minimize false positives
P-values interpretable
Context of discovery
Pushes knowledge into new areas/ predata-led discovery
Finds unexpected relationships
Goal is to minimize false negatives
P-values meaningless
“In statistics, hypotheses suggested by a given dataset, when tested with the same dataset that suggested them, are likely to be accepted even when they are not true. This is because circular reasoning (double dipping) would be involved: something seems true in the limited data set, therefore we hypothesize that it is true in general, therefore we (wrongly) test it on the same limited data set, which seems to confirm that it is true. Generating hypotheses based on data already observed, in the absence of testing them on new data, is referred to as post hoc theorizing (from Latin post hoc, "after this").
The correct procedure is to test any hypothesis on a data set that was not used to generate the hypothesis.”
Presenting exploratory results as confirmatory increases publishability at the expense of credibility

43. Preregistration or Replication
Level 1: Disclose preregistration, encourage replication

44. How Stop HARKING? Pre-registration / Registered reports
“Because the study is accepted in advance, the incentives for authors change from producing the most beautiful story to producing the most accurate one.”
Chris Chambers, Cardiff University (section editor at Cortex and Royal Society Open Science)
“Registered Reports eliminates the bias against negative results in publishing because the results are not known at the time of review”
Daniel Simons, University of Illinois (co-Editor of Registered Replication Reports at Perspectives on Psychological Science)

45. Preregistration or Replication
Level 3: Registered Reports
If YES, then study is granted “in principle acceptance” (IPA), a promise to publish regardless of outcome.
Are the hypotheses well founded and worth addressing?
Are the methods and proposed analyses able to address the hypotheses?
Have the authors included sufficient positive controls to confirm that the study will provide a fair test?

46. Preregistration or Replication
Level 3: Registered Reports
Did the authors follow the approved protocol?
Did positive controls succeed?
Are the conclusions justified by the data?
88 Journals use Registered Reports, see more at cos.io/rr

47. Materials transparency Data transparency Code transparency
Level 1 Level 2 Level 3
Eight Standards
Data citation
Materials transparency
Data transparency
Code transparency
Design transparency
Study Preregistration
Analysis Preregistration
Replication
What does it mean?
What does it solve?
How do we get authors to do it?
See cos.io/top for more detailed language

48. How can you make researchers do better. Journals
How can you make researchers do better? Journals. How can you make JOURNALS do better? 1) Make it easier for them to do it. COS

49. 2) Peer Pressure....

50. How can you make researchers do better?
Journals can help.
How can you make JOURNALS do better?
Make it easier for them to do it. COS
Peer pressure.
Organizational action.
Individual editor action.
Pressure from reviewers.
Pressure from researchers (readers, authors).

51. Advances in Methods and Practices in Psychological Science
Or – Start a New Journal
Advances in Methods and Practices in Psychological Science

52. How...
...can journals help document the problems? Raise awareness of the problems?
...did journals contribute to the problems?
...can journals help to reduce the problems in the future?
...can journals help fix the problems they created in the past (i.e., fix the record)?

53. What Is NOT Happening? Building Better Bricks but not Better Buildings
1. Stop losing important information. 2. Get better at compiling results. (Meta-analysis) 3. Get better at connecting findings. keywords (ugh) reasons for citations (like law?) More Theory / Review Journals ? 5. More open places for (moderated?) commentary / discussion for post-pub discussion

54. Why are there failures to replicate?
Fault of Original Researcher
Fault of Neither, One, or Both
Fault of Replicator
Fraud
QRPs (“Questionable Research Practices”)
Chance
Bad Copy (methods / data / analysis description)
QRPs
Bad intentions
Incompetence

55. “A success of using technology and changing incentives to implement the timeless values of science.”

56. Advances in Methods and Practices in Psychological Science
END
Thanks to...
Brian Nosek
Simine Vazire
Alison Ledgerwood
David Mellor
Alan Kraut
The Center for Open Science
Advances in Methods and Practices in Psychological Science

57. QRPs: Questionable Research Practices

59. 1 2 3 4 5
Include/Exclude participants to achieve p<.05.
Collect and analyze multiple conditions, drop those that do not show p<.05.
Stop collecting data once p<.05 is reached (or keep collecting more data until p<.05).
Include many measures, but report only those p<.05.
Include covariates in statistical analysis to get p<.05.