1. The NIH Peer Review Process Sally A. Amero, Ph.D. NIH Review Policy Officer Office of Extramural Research 2010 NIH Regional Seminars

2. 22 The NIH Peer Review Process Two-tiered Process Mandated by law – PHS Health Act Defined in federal regulation – 42 CFR 52h Further defined in NIH policy Per year: – Nearly 80,000 applications – Over 18,000 reviewers

3. 33 The NIH Peer Review Process Initial peer review – Recommendations on scientific and technical merit – Scientific Review Groups (SRGs or “Study Sections”) Advisory Council or Board – Recommendations to the ICs on funding, appeals, program priorities – “Council” Final funding decisions – IC Director

4. 44 The NIH Peer Review Process Application received – CSR* Assignments made  Initial peer review Funding considerations SRG; study section Institutes or Centers (ICs) IC or CSR*Duals possible Scientific Review Officer Program Officer  Second level of review  Funding decisions Council or Board (IC)IC Director  Award! * CSR = NIH Center for Scientific Review

5. 55 The NIH Peer Review Process CSR Review Most R01s, fellowships, and small business applications Some Program Announcements (PAs, PARs) Some Requests for Applications (RFAs) Institute/Center Review IC-specific features Program projects Training grants Career development awards RFAs Referral to an SRG The review locus is stated in the Funding Opportunity Announcement.

6. 66 The NIH Peer Review Process To Request a Scientific Review Group Cover letter of application – Application title – FOA # and title – Request: – Assignment to particular SRG or study section – Assignment to particular IC for funding consideration – Disciplines involved, if multidisciplinary – Explanation for late application Not all requests can be honored. SRG rosters are posted 30 days before the SRG meeting: http://era.nih.gov/roster/index.cfm http://www.csr.nih.gov/committees/rosterindex.asp

7. 77 The NIH Peer Review Process Scientific Review Officer (SRO) First level of peer review – Designated Federal Official – Extramural scientist administrator – Identifies and recruits reviewers – Manages conflicts of interest – Oversees arrangements for review meetings – Presides at review committee meetings – Prepares and releases summary statements

8. 88 The NIH Peer Review Process Peer Reviewers Recruitment – Expertise – Stature in field – Mature judgment – Impartiality – Ability to work well in a group – Managed conflicts of interest – Balanced representation Gender Geography Diversity Seniority – Availability

9. 99 The NIH Peer Review Process Types of Scientific Review Groups (SRGs) “Chartered” SRGs – Multiyear terms – Formal appointment process – May include temporary members for special expertise Special Emphasis Panels (SEP) – Ad hoc membership – Often meet only once

10. 10 The NIH Peer Review Process Types of Reviewers Regular reviewers – permanent and temporary – Preliminary impact/priority scores, criterion scores, written critiques – Final impact/priority scores Other Contributing Reviewers (“mail” reviewers) – Written critiques, criterion scores, preliminary impact/priority scores – Cannot submit final impact/priority scores

11. 11 The NIH Peer Review Process Reviewer Assignments For each application: – ≥ Three qualified reviewers are assigned (“2 + 1”) – Assignments are made by the SRO Based on the scientific content of application Expertise of the reviewer Suggestions from the PI on types of expertise – not names! Suggestions from Program staff Suggestions from SRG members Managing conflicts of interest Balancing workload Assignments are confidential!

12. 12 The NIH Peer Review Process Conflicts of Interest (COI) – Financial – Employment – Personal – Professional – SRG membership – Other interests Two COI vouchers are submitted by each SRG member. COI between a reviewer and an application:

13. 13 The NIH Peer Review Process Scientific Review Groups (SRGs) Make recommendations on merit - not funding! – Scientific and technical merit – Budget and project duration – Protection of human subjects, inclusion plans, vertebrate animals, biohazards – Resource Sharing Plans – Other administrative factors  Impact/priority scores  Criterion scores  Written critiques

14. 14 The NIH Peer Review Process Confidentiality All confidential materials, discussions, documents are deleted, retrieved, or destroyed Reviewers sent guidance with applications Application information provided on secure websites or protected portable devices All questions must be referred to SRO SRG meetings are closed to the public Program staff may observe SRG meeting Do not contact reviewers directly!

15. 15 The NIH Peer Review Process Overall Impact: Likelihood for the project to exert a sustained, powerful influence on the research field(s) involved In consideration of: – At least five scored criteria Receive individual, numerical scores Additional criteria in certain announcements – Additional review criteria As applicable for the project proposed Do not receive individual, numerical scores Additional criteria in certain announcements

16. 16 The NIH Peer Review Process Scored Review Criteria Applications for: – Research grants – Cooperative agreements Other criteria apply to other mechanisms Significance Investigator(s) Innovation Approach Environment (FOA-specific criteria) See “Review Criteria at a Glance” (http://grants.nih.gov/grants/peer/reviewer_guidelines.htm)

17. 17 The NIH Peer Review Process Scored Review Criteria Significance Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

18. 18 The NIH Peer Review Process Scored Review Criteria Investigator(s) Are the PD/PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?

19. 19 The NIH Peer Review Process Scored Review Criteria Innovation Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?

20. 20 The NIH Peer Review Process Scored Review Criteria Approach Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?

21. 21 The NIH Peer Review Process Scored Review Criteria Approach If the project involves clinical research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed?

22. 22 The NIH Peer Review Process Scored Review Criteria Environment Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?

23. 23 The NIH Peer Review Process Additional Review Criteria As applicable for the project proposed, reviewers: Consider in determining scientific and technical merit Do not give separate scores for these items. FOA-specific criteria Protections for Human Subjects Inclusion of Women, Minorities, and Children Vertebrate Animals Resubmission Applications Renewal Applications Revision Applications Biohazards

24. 24 The NIH Peer Review Process Additional Review Considerations As applicable for the project proposed, reviewers: Address each item Do not give scores for these items Should not consider them in providing an overall impact/priority score. FOA-specific considerations Select Agent Research Applications from Foreign Organizations Resource Sharing Plans Budget and Period Support

25. 25 The NIH Peer Review Process NIH Scoring System Numerical scores – 1.0 (exceptional) to 9.0 (poor) – Final impact/priority score - average of individual scores x 10 – Individual criterion scores – Ranked by percentile for certain mechanisms – Not Discussed (ND) - streamlining – Other designations (NR, DF, AB, NP, etc.) Final impact/priority scores range from 10 through 90.

26. 26 The NIH Peer Review Process NIH Scoring System Preliminary scores (before the SRG meeting) – Entered by assigned reviewers and discussants in secure website – Made available to other SRG members Final overall impact/priority scores (at the SRG meeting) – Voted by private ballot – All eligible SRG members vote Reviewers are instructed to revise their criterion scores after the meeting.

27. 27 The NIH Peer Review Process Score Descriptors Phases of Process ImpactScoreDescriptor High Impact 1Exceptional 2Outstanding 3Excellent Moderate Impact 4Very Good 5Good 6Satisfactory Low Impact 7Fair 8Marginal 9Poor

28. 28 The NIH Peer Review Process Streamlining Allows discussion of more meritorious applications – Less meritorious applications tabled at the – SRG meeting, designated Not Discussed (ND) – Requires full concurrence of the entire SRG – Summary statement: Reviewer critiques Individual criterion scores No numerical, overall impact/priority score

29. 29 The NIH Peer Review Process Streamlining Score order of review – SRG discusses most meritorious applications first – Entire SRG decides when to stop, which applications will not be discussed in panel Other order of review (e.g., IC assignment, mechanism) – SRO prepares a list of average preliminary scores – Distributes to SRG – Entire SRG decides which applications to discuss

30. 30 The NIH Peer Review Process Pre-Meeting SRG Procedures SRO – Performs administrative review of applications – Recruits reviewers, arranges for meeting date and site – Assigns 3 SRG members to each application – Makes applications available to reviewers Internet Assisted Review (IAR) site or on CDs Usually about six weeks before the SRG meeting – Instructs reviewers in review procedures – Monitors posting of initial scores and critiques in IAR Documents for Reviewers are available at: http://grants.nih.gov/grants/peer/reviewer_guidelines.htm#general_guidelines

31. 31 The NIH Peer Review Process Structured Critiques Phases of Process New summary statement format –Bulleted comments from reviewers, less text –Criterion scores from assigned reviewers –Decreases variability –Increases quality of information in critiques –More succinct, better organized –Encourages evaluative statements –Ensures that reviewers address all review criteria and considerations Critique templates are available at: http://grants.nih.gov/grants/peer/reviewer_guidelines.htm#general_guidelines

32. 32 The NIH Peer Review Process Templates for Reviewers Phases of Process Links to definitions of review criteria

33. 33 The NIH Peer Review Process Pre-Meeting SRG Procedures Reviewers – Examine assignments – Submit Conflict of Interest and Confidentiality voucher – Read applications, prepare written critiques in templates – Enter preliminary scores into IAR – Read and consider other critiques and preliminary scores – Make travel and hotel arrangements Preliminary scores and critiques may be due several days or a week in advance!

34. 34 The NIH Peer Review Process Post-submission Materials Applications submitted for Sept. 25 th, 2010 and later: – Will only accept administrative materials resulting from unanticipated events, such as – Revised budget page(s) (e.g., due to new funding) – Biographical sketches (e.g., due to the loss of an investigator) – Letters of support or collaboration (e.g., due to the loss of an investigator) – News of an article accepted for publication Special provisions for training grants and certain FOAs

35. 35 The NIH Peer Review Process SRG Meetings Agenda – Call to Order - Chairperson – Policy and instructions - SRO – Discuss applications one at a time – Where feasible: In score order Cluster New Investigator (NI) applications Cluster clinical applications – Score each application by private ballot after its discussion – Discuss other considerations Budget Resource Sharing Plans

36. 36 The NIH Peer Review Process SRG Meeting Procedures Discussion format – Members with conflicts excused – Initial levels of enthusiasm stated (assigned reviewers and discussants) – Primary reviewer - explains project, strengths, weaknesses – Other assigned reviewers and discussants follow – Open discussion (full panel) – Levels of enthusiasm (assigned reviewers) re-stated – Individual SRG members vote – Other review considerations discussed (budget)

37. 37 The NIH Peer Review Process SRG Meeting Procedures If 60 applications/SRG meeting ~ 50% streamlined, 30 applications to discuss and score Leaves ~ 14 minutes on average/application ~ 3 - 4 minutes/reviewer If 9 hour SRG meeting ~ ½ hour introduction, streamlining ~ 1 hour lunch, 2 x 15 minute breaks Clarity and brevity are essential!

38. 38 The NIH Peer Review Process After the Review eRA Commons (http://era.nih.gov/commons/index.cfm)http://era.nih.gov/commons/index.cfm – Final Impact/Priority Score available three days after the SRG meeting – Summary statement available 4 – 8 weeks after meeting Available also to Program Officers at that time Confidential document Available to: – PD/PIs – NIH officials – Advisory Council members

39. 39 The NIH Peer Review Process Summary Statement First page – NIH Program Officer (upper left corner) Name Contact information – Final Impact/Priority Score or other designation – Percentile (if applicable) – Codes Human subjects Vertebrate animals Inclusion plans – Budget request A favorable score does not guarantee funding!

40. 40 The NIH Peer Review Process Summary Statement - continued Subsequent Pages – Description (provided by applicant) – Resumé and Summary of Discussion (if discussed) – Reviewer critiques – essentially unedited Follow review criteria for mechanism Protections for Human Subjects Inclusion Plans Vertebrate Animals Biohazards Budget – Administrative Notes – Meeting roster

41. 41 The NIH Peer Review Process Appeals Process NIH Program Officer = Point of Contact – If the outcome is unfavorable, consider your options: Revise and resubmit application – Consider critiques in summary statement – Address critiques in introduction and text Appeal the review outcome – Procedural deficiencies – Factual errors – May result in re-review of same application by the same or different SRG Discuss your options with your Program Officer!

42. 42 The NIH Peer Review Process Appeals Process Unresolved appeals are presented to the IC Council – Council options: Support the SRG review Support the appeal, recommend a re-review – Application could be deferred for next round – Application cannot be modified or updated – Results of a re-review cannot be appealed further – Council cannot overturn the SRG review or impact/priority score

43. 43 The NIH Peer Review Process Advisory Council/Board Second level of review – recommendations on: – Research priority areas – Policy – Appeals – Funding – Quality of SRG review Members – Scientists from the extramural research community – Public representatives – Appointed to multi-year terms – Appointed as Special Government Employees

44. 44 The NIH Peer Review Process Advisory Council/Board Balanced representation – Expertise – Stature in field – Mature judgment – Impartiality – Managed conflicts of interest – Balanced representation Gender, Diversity Geography, Seniority

45. 45 The NIH Peer Review Process Funding Considerations Authority of the IC Director – Scientific and technical merit (initial peer review) – Council recommendations – Relevance to program priorities in IC – Compliance with policies – Number of meritorious applications received – Availability of funds – Advice of IC Program Staff

46. 46 The NIH Peer Review Process Enhancing Peer Review Initiative http://enhancing-peer-review.nih.gov/ Office of Extramural Research Peer Review Process http://grants.nih.gov/grants/peer_review_process.htm Peer Review Policies & Practices http://grants.nih.gov/grants/peer/peer.htm Center for Scientific Review http://cms.csr.nih.gov/AboutCSR/Welcome+to+CSR/ Additional Information

47. 47 The NIH Peer Review Process Contact Information Sally Amero, Ph.D. NIH Review Policy Officer Extramural Research Integrity Liaison Officer Office of Extramural Programs Office of Extramural Research National Institutes of Health ameros@od.nih.gov