The Ins and outs of nih peer review Rebecca Wagenaar-Miller, PhD Scientific Review Officer National Institute of Dental and Craniofacial Research NIH

Context-My Background PhD Vanderbilt University Department of Cancer Biology (2002) Matrix Metalloproteinase-7 in Mouse Models of Intestinal Tumorigenesis Vanderbilt University Postdoctoral Fellow (2002-2003) MMP-7 in pre-clinical model of colorectal cancer NIH, NIDCR Postdoctoral Fellow (2003- 2007) Type II transmembrane serine proteases MMP-mediated and uPARAP-mediated collagen degradation in development Scientific Review Officer (SRO), NIDCR, NIH (2007-present)

Goals Give a brief look into the NIH peer review process Update you on changes in NIH peer review Discuss strategies for a successful application Provide a few resources

Is this your view of the NIH?

How about this?

Or this?

Research grant Review process National Institutes of Health Research Grant Application School or Other Research Center Center for Scientific Review Assigns to IRG/Study Section & IC Study Section Initiates Research Idea Submits Application Evaluates for Scientific Merit Institute Evaluates for Program Relevance Advisory Councils and Boards Allocates Funds Conducts Research Recommends Action Institute Director Takes final action for NIH Director

The peer review process

THE MISSION OF PEER REVIEW To ensure that grant applications and contract proposals are evaluated by qualified scientific peers in a fair, objective, and timely manner.

The NIH peer review process Conducted according to the Federal Advisory Committee Act Meetings are closed to the public All materials and discussions – strictly confidential Over 80,000 applications reviewed per year Almost 18,000 reviewers GOALS

NIH Application review 80% of applications are reviewed by CSR in Scientific Review Groups (SRGs) that are part of Integrated Review Groups (IRGs) The rest are reviewed by individual ICs Requests for Applications (RFAs) Program Announcements with IC Review (PARs) Training Applications (F, K’s) ‘Complex’ Applications (P, U’s, Clinical Trials) Contract Proposals

Summary statements The legal summary of the review meeting Includes all three reviewer critiques, largely unedited Includes a resume, which is a summary of the discussion of the meeting, focusing on the major strengths and weaknesses that resulted in the overall impact score Used by council, applicants, program staff

grants and peer review life cycle Award and Post Award Pre-application Peer Review Council Program Scientific Review Officer Program and GM Peer Review Meeting Applications Arrive Summary Statement Council 2-3 months 2-6 weeks One month

The NIH Peer Review Process To Request a Scientific Review Group Cover letter of application Application title FOA# and title Request: Assignment to particular SRG or study section Assignment to particular IC for funding consideration Disciplines involved, if multidisciplinary Explanation for late application Not all requests can be honored. SRG rosters are posted 30 days before the SRG meeting: http://era.nih.gov/roster/index.cfm http://www.csr.nih.gov/committees/rosterindex.asp

The NIH Peer Review Process Reviewer Assignments For each application: ≥ Three qualified reviewers are assigned (“2 + 1”) Assignments are made by the SRO Based on the scientific content of application Expertise of the reviewer Suggestions from the PI on types of expertise – not names! Suggestions from Program staff Assignments are confidential!

9 Point rating scale Impact Score Descriptor Additional Guidance on Strengths and Weaknesses High Impact 1 Exceptional Exceptionally strong with essentially no weaknesses 2 Outstanding Extremely strong with only negligible weaknesses 3 Excellent Very strong with only some minor weaknesses Moderate Impact 4 Very Good Strong but with numerous minor weaknesses 5 Good Strong but with at least one moderate weakness 6 Satisfactory Some strengths but also some moderate weaknesses Low Impact 7 Fair Some strengths but with at least one major weakness 8 Marginal A few strengths and a few major weaknesses 9 Poor Very few strengths and numerous major weaknesses Non-numeric score options:  NR = Not Recommended for Further Consideration, DF = Deferred, AB = Abstention, CF = Conflict, NP = Not Present, ND=Not Discussed Minor Weakness: An easily addressable weakness that does not substantially lessen impact Moderate Weakness: A weakness that lessens impact Major Weakness: A weakness that severely limits impact 1 3 2

The NIH Peer Review Process NIH Scoring System Preliminary scores (before the meeting) Entered by assigned reviewers and discussants in secure website Made available to other SRG members Final overall impact/priority scores (at the SRG meeting) Voted by private ballot All eligible members vote Reviewers are instructed to revise their criterion scores after the meeting.

The NIH Peer Review Process Post-submission Materials Applications submitted for Sept. 25th, 2010 and later: Will only accept administrative materials resulting from unforeseen administrative issues, such as Revised budget page(s) (e.g., due to new funding) Biographical sketches (e.g., due to the loss of an investigator) Letters of support or collaboration (e.g., due to the loss of an investigator) Adjustment resulting from natural disaster (e.g., loss of animal colony) Adjustments resulting from change of institution (e.g., PI moves to another university) News of an article accepted for publication (do not sent article) Special provisions for training grants and certain FOAs Notice OD-10-115

The NIH Peer Review Process SRG Meeting Procedures Discussion format Members with conflicts excused Initial levels of enthusiasm stated(assigned reviewers and discussants) Primary reviewer - explains project, strengths, weaknesses Other assigned reviewers and discussants follow Open discussion (full panel) Levels of enthusiasm (assigned reviewers) restated Individual SRG members vote Other review considerations discussed (budget)

Peer review changes

New NIH Application Formats NEW Sections OLD Sections INTRODUCTION-Revision (1 page) INTRODUCTION-Resubmission (3 pages) SPECIFIC AIMS (1 page) RESEARCH STRATEGY (R01-12 PAGES) R01 25 pages Significance Background and Significance Innovation Approach- Research Design and Methods -(preliminary studies-part of approach) Preliminary Studies -(progress report-part of approach) Progress report

Scoring and review criteria Research Project Grants (R01, R21, R03) Fellowship (F30, F32, F32, F33) Career Development (K02, K08, K23, K25, K99) OVERALL Overall Impact Overall Impact/Merit Scored Review Criteria Significance Investigator(s) Innovation Approach Environment Fellowship Applicant Sponsors, Collaborators, and Consultants Research Training Plan Training Potential Institutional Environment & Commitment to Training Candidate Career Development Plan/Career Goals & Objectives/Plan to Provide Mentoring Research Plan Mentor(s), Co-Mentor(s), Consultant(s), Collaborator(s), Environment & Institutional Commitment to the Candidate Additional Protections for Human Subjects Inclusion of Women, Minorities, & Children Vertebrate Animals Biohazards Resubmission Renewal Revision Consider- ations • Applications from Foreign Organizations • Select Agents • Resource Sharing Plans Budget & Period of Support Training in the Responsible Conduct of Research Applications from Foreign Organizations Select Agents Resource Sharing Plans Impact Score Do not Impact Score

Overall impact/priority score Research Project Grant (RPG): likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the five core review criteria, and additional review criteria Fellowship (F): likelihood that the fellowship will enhance the candidate’s potential for, and commitment to, a productive independent scientific research career in a health-related field, in consideration of the scored and additional review criteria Career Development (K): likelihood for the candidate to maintain a strong research program, taking into consideration the criteria in determining the overall impact/priority score

New Investigators and EARLY STAGE INVESTIGATORS

New investigator Background: Goals: Concern about the number of New Investigators (NI) and the increasing age at first award Goals: Enrich the applicant pool with enough Early Stage Investigator (ESIs) to reduce the average age at first award Accelerate the period of training leading to independence Background: NIH has a long standing commitment to training and research funding of new investigators, understanding that they are the innovators of the future, bringing fresh ideas and technologies to existing biomedical research problems, and pioneering new areas of investigation.

Early Stage Investigators NIH created a new ‘Early Stage Investigator’ (ESI) category designed to accelerate the early transition of new scientists to research independence by receiving their first R01 earlier. http://grants.nih.gov/grants/guide/notice- files/NOT-OD-08-121.html A PD/PI who qualifies as a New Investigator is considered an Early Stage Investigator (ESI) if he/she is within 10 years of completing his/her terminal research degree or is within 10 years of completing medical residency (or the equivalent). 

New investigator/early stage investigator New Investigator Policies are limited to applications for traditional research project grant (R01) support. All New Investigators are encouraged to apply for NIH R01 awards. During Peer Review: NI/ESI applications will be clustered during review Consider the career stage of the PD/PI when evaluating elements of the applications, such as the availability of preliminary data NOT-OD-08-121 September 26, 2008

Implementation of ESI Definition The NIH modified the collection of information on degree dates and medical residency within the personal profile of the eRA Commons. PD/PIs must update their personal profile in the eRA Commons in order to be considered for the ESI classification.  Investigators who enter degree and residency completion dates will be notified of their ESI status by email. A procedure and guidelines for requesting an extension of the period of ESI eligibility is in place to accommodate individuals with various medical concerns, disability, pressing family care responsibilities, or active duty military service (instructions in Commons).

NI AND ESI POLICY FY 2010 Over the past three years, about 25 percent of all competing R01 awards have gone to New Investigators Continue to equilibrate success rates for established and New Investigators submitting new R01 applications Should permit the NIH to support 1,650 or more New Investigators A majority of the New Investigators will be Early Stage Investigators http://grants.nih.gov/grants/guide/notice-files/NOT-OD-10-039.html NOT-OD-10-039,January 8, 2010

http://grants.nih.gov/grants/oer.htm

Tips for a successful grant application

Grant Writing for Success Start early Start with a good idea Talk to your NIH Program Official(s) Remember review criteria Show you draft application to colleagues who do not already know what you intend to do who are not your best friend who has/had the same type of grant Follow instructions carefully 32 32

Foa SOURCE OF REVIEW GUIDELINES

Make YOUR strengths the focus of your application Propose experiments that take advantage of YOUR training YOUR expertise YOUR environment Don't say that you're going to learn a sophisticated new technique and then apply it in later years If you need a collaborator(s), get a strong one who publishes in peer-reviewed journals and who has a grant

Focus! A major problem with applications from first time investigators is being overly ambitious. Focus Did I remember to say “Focus”? Be certain every aim and experiment is clearly related to the overall goal of the application.

grants1.nih.gov/grants/guide/listserv.htm

Insider tips and tricks Late submission policy for NIH study section service for any PI/PD of grant application (NOT-OD-08-027) Standard receipt dates (1 or 2 weeks late) “Late applications have been accepted for reasons such as…temporary or ad hoc service on a NIH extramural peer review group” “No NIH staff member whether in the Center for Scientific Review or any of the other Institutes/Centers has the authority to give permission in advance for a late application.” Cover letter important Identify study section Identify areas of expertise needed Can list conflicts that should not review application (but be prepared to justify) Submit to R01 to take advantage of NI/ESI mandate Submit to RFAs if possible… free review of application

Research Portfolio Online Reporting Tool (RePORT) http://report. nih A Searchable database of federally supported biomedical research Access reports, data, analyses, expenditures, results of NIH supported research activities Identify, Analyze IC(s) research portfolios, funding patterns, funded investigators: Identify areas with many or few funded projects Identify NIH-funded investigators and their research Identify potential mentors/collaborators 38

Websites NIH Guide: http://grants.nih.gov/grants/guide/ NIH Office of Extramural Research http://grants.nih.gov/grants/oer.htm NIAID “how to” website for developing a grant application: http://www.niaid.nih.gov/ncn/grants/default.htm Writing a Grant: A Technical Checklist from NINDS: http://www.ninds.nih.gov/funding/grantsmanship_checklist.htm  NIH RePorter: database of all funded grants including abstracts, funding amounts, study section, PI, and much more info: http://projectreporter.nih.gov/reporter.cfm NIH Peer Review Policies and Practices http://grants.nih.gov/grants/peer/peer.htm Changes to Peer Review (and applications) http://enhancing- peer-review.nih.gov/docs/application_changes.pdf

Thanks