1. Luci Roberts, Director of Planning and Evaluation Katrina Pearson, Assistant Director, Division of Statistical Analysis and Reporting Sally Amero, NIH Review Policy Officer Office of Extramural Research, NIH

2. Rationale Unrelenting Demands on Peer Review Number of applications continues to increase Number of grantees stays the same 60% of NIH-funded investigators have served in NIH peer review in the last ten years 35-40% of NIH-funded investigators served in each of FYs 2011 - 2014 Question: How can we continue to uphold the excellence of peer review while faced with increased numbers of applications and the same or fewer reviewers? 2

3. Approach Question 1: What is a reasonable expectation for review service from NIH-funded investigators? Surveys of SROs, reviewers and prospective reviewers to determine factors that contribute to: SRO’s choices about whom to invite Reviewers choices about whether to accept 3

4. Approach Question 2: Which NIH–funded investigators serve as reviewers? Using the IMPAC2 Person data, compare the characteristics of NIH-funded investigators who have served as reviewers to those who have never served, examine whether their characteristics have changed over time. 4

5. Question 1 What is a reasonable expectation for review service from NIH-funded investigators? Survey Inclusion Criteria Applicant-reviewers: (n = ~4,000; 1830 total respondents; 46%) – submitted at least one grant application as single PI or contact PI in the past five years (any activity code) – had active funding from NIH in past five years (any activity code, including F, T, K, etc.) SROs : (n = 423; 271 respondents; 64%) - All SROs who had a SRG assigned to them during the two rounds prior to the survey 5

6. Respondent Counts NIH Grantees – highest role on project (past 5 years) – Principal Investigators/Project Directors: 1,616 – Subproject/Core/Consortium Leads: 37 (402 total) – Training Program Directors: 8 (145 total) Have worked on NIH Clinical Research in past 5 years (Defined in the Q as Research involving Human Subjects) – Yes: 754 (43%) – No: 1,011 6

7. 7

8. During the initial phase of your effort to recruit reviewers,… 8

9. Highest Project Roles of respondents asked to review (and not asked) 9

10. Ranks of PI/PD respondents asked to review (and not asked) 10 Asked to ReviewNot Asked to Review

11. When you reviewed in the past year, was the review process more burdensome than it could be? (and if yes, how so?) Reasons given for “yes”: Number of applications assigned: 45% Too much time devoted to applications that will never be awarded: 16% Applications assigned to reviewers that do not have appropriate expertise: 5% Too many other demands on time: 5% Travel/meeting accommodations: 4% 11

12. 12

13. 13

14. 14

15. What the surveys found…. Reviewers: time required to prepare reviews is most burdensome; most widely selected reason to decline an invitation Reviewers: Assistant Professors share similar motivations as more senior investigators; grantsmanship experience is important at all levels NIH’s service expectations exceed the reported capacity of most reviewers to serve More than half of the NIH respondents to the survey had not been invited to review Continuous submission and Flexible terms of service not a proximal influence on the decision to accept an invitation to review SROs have a good sense of what motivates reviewers to review: – responsibility, – social networking, and – improved knowledge of grantsmanship and the science in their field SROs rely primarily on Principal Investigators, fewer SROs recruit from the other roles 15

16. Based on the survey results… Clear guidance on eligibility criteria for ad hoc service may be helpful – particularly criteria that do not narrow the pool of eligible reviewers as success rates shrink Better (searchable) access to information on investigators involved in other roles on funded grant may be helpful to SROs searching for reviewers Information about who has been invited to review would be really helpful for understanding how to improve NIH’s capacity for Peer Review 16

17. Characteristics of NIH Reviewers: Question 2 Which NIH–funded investigators serve as reviewers? 17

18. Data and Methods  Constructed a sampling frame from IMPAC II, FY 2009 -2014 containing all persons who submitted a competing grant application  Sub-analyses restricted reviewers to those with R01-equivalent or “Other” grant support excluding fellowship and career awards  ARRA Applications were excluded  Funding amounts for multiple PI awards for apportioned equally to each of the multiple PIs 18

19. R01-Equivalent Awardees = 25,294 Other Awardees = 18,998 19 4,069 PIs total 1,586 did not review 8,751 PIs total 2,013 did not review

20. R01-Equivalent Awardees = 25,294 Other Awardees = 18,998 20 5,741 PIs total 2,101 did not review 8,340 PIs total 1,576 did not review

21. In Conclusion – Investigators with the least amount of support served at a lower rate – Most investigators who have had R01-equivalent awards during the 5-year period have served as reviewers – However, a significant number of PIs with > $500K in recent R01 support have not served as peer reviewers – NIH Issued NOT-OD-15-035.html:NOT-OD-15-035.html “The NIH expects principal investigators of NIH supported grants and contracts to serve on NIH peer review groups, when asked… Therefore, the NIH expects grantee institutions and R&D contract recipients to encourage their NIH-funded investigators to serve on NIH peer review and advisory groups. ” 21

22. Modeling Exercise Assume that, in response to a large influx of new applications, NIH needs to recruit more reviewers to participate in peer review. How many more reviewers could be enlisted from our own PI ranks? Assumptions: – Seek reviewers with at least one active award, at least $500k in funding, and at least one R01 award in the past five years – 80% of PIs who have not served agree to review for one meeting each – 6 applications per reviewer/meeting on average 22

23. 23

24. Acknowledgements Division of Statistical Analysis and Reporting http://grants.nih.gov/grants/oer_offices/subopac_dsar.htm http://grants.nih.gov/grants/oer_offices/subopac_dsar.htm Robin Wagner, Ph.D. Matthew Eblen Deepshika Roychowdhury Sam Shuptrine Division of Planning and Evaluation Joy Wang, Ph.D. Pritty Joshi, Ph.D. Research Triangle Institute, International 24