Role of Peer Review in Solving the Problem of Reproducibility American Physiological Society 2015 Richard Nakamura, Ph.D. Director NIH Center for Scientific Review

NIH . . . Turning Discovery Into Health NIH’s mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability. NIH achieves its mission largely through awarding research grants based upon peer review of applications from extramural scientists

CSR Peer Review – Fiscal Year 2014 86,000 applications received 16,000 reviewers 237 Scientific Review Officers 1,500 review meetings

NIH budget doubling ended in 2003

NIH Program Level in Nominal Dollars and Constant 1998 Dollars, FY1998 – FY2014 (In Billions) Appropriation Appropriation in 1998 Dollars

Number of Applications Received by Fiscal Year FY 2013 ~ 83,800, FY 2014 ~ 86,540, a 3.3% increase (as of 10/19/2014).

We are caught in a vicious cycle of dropping success rates (award rates) and increasing applications

Scientists struggling to stay ahead in an arms race of intense competition are amplifying their story lines and weakening their self checks, while constantly searching for more funding.

The number of applications and low paylines has had a palpable effect on the perceived quality of peer review.

In this environment our science cannot afford the economic waste of irreproducible results, let alone any loss of confidence in our product by the nation.

How to intervene The most important intervention point is at journal review of research reports. Here is where the exact study design and methods can be strongly evaluated and where many editors have agreed to better reporting of methods including reporting of statistical power. Here is where electronic space can be created to allow archival publication of methodologic details and data can be shared so studies can be more easily reproduced.

Is the grant application an appropriate intervention point?

The conservatism problem For decades we have assumed that funded scientists will employ rigorous methods and that grant applications (particularly in basic areas) will reflect work in progress and general directions, rather than a fixed protocol. Funding institutes already complain about the conservatism of peer review due to prioritization of approach. Further attending to rigor will increase conservatism. NIH has determined that until we collectively turn around the problem of reproducibility, review conservatism will have to be accepted. CSR will therefore place additional emphasis on premise and design rigor until the lesson is established.

NIH has not yet finalized the language providing guidance to reviewers NIH has not yet finalized the language providing guidance to reviewers. This should be resolved this month.

In the meantime, 1) please adopt the best practices discussed here in your experiments and 2) help rally national understanding of a science and technology sector funding crisis.

Questions? CSRDirector@csr.nih.gov