1. Role of Peer Review in Solving the Problem of Reproducibility
American Physiological Society 2015
Richard Nakamura, Ph.D.
Director
NIH Center for Scientific Review

2. NIH . . . Turning Discovery Into Health
NIH’s mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability.
NIH achieves its mission largely through awarding research grants based upon peer review of applications from extramural scientists

3. CSR Peer Review – Fiscal Year 2014
86,000 applications received
16,000 reviewers
237 Scientific Review Officers
1,500 review meetings

4. NIH budget doubling ended in 2003

5. NIH Program Level in Nominal Dollars and Constant 1998 Dollars, FY1998 – FY2014
(In Billions)
Appropriation
Appropriation in 1998 Dollars

6. Number of Applications Received by Fiscal Year
FY 2013 ~ 83,800, FY 2014 ~ 86,540, a 3.3% increase (as of 10/19/2014).

7. We are caught in a vicious cycle of dropping success rates (award rates) and increasing applications

8. Scientists struggling to stay ahead in an arms race of intense competition are amplifying their story lines and weakening their self checks, while constantly searching for more funding.

9. The number of applications and low paylines has had a palpable effect on the perceived quality of peer review.

10. In this environment our science cannot afford the economic waste of irreproducible results, let alone any loss of confidence in our product by the nation.

11. How to intervene
The most important intervention point is at journal review of research reports.
Here is where the exact study design and methods can be strongly evaluated and where many editors have agreed to better reporting of methods including reporting of statistical power.
Here is where electronic space can be created to allow archival publication of methodologic details and data can be shared so studies can be more easily reproduced.

12. Is the grant application an appropriate intervention point?

13. The conservatism problem
For decades we have assumed that funded scientists will employ rigorous methods and that grant applications (particularly in basic areas) will reflect work in progress and general directions, rather than a fixed protocol.
Funding institutes already complain about the conservatism of peer review due to prioritization of approach.
Further attending to rigor will increase conservatism.
NIH has determined that until we collectively turn around the problem of reproducibility, review conservatism will have to be accepted.
CSR will therefore place additional emphasis on premise and design rigor until the lesson is established.

14. NIH has not yet finalized the language providing guidance to reviewers
NIH has not yet finalized the language providing guidance to reviewers. This should be resolved this month.

15. In the meantime, 1) please adopt the best practices discussed here in your experiments and 2) help rally national understanding of a science and technology sector funding crisis.

16. Questions? CSRDirector@csr.nih.gov