NIH Review Procedures Betsy Myers Hospital for Special Surgery

General Process for Proposal Review Applicant has idea Forms are submitted Proposal is reviewed for scientific merit Summary statement is prepared Funding Institute or Center gives information to applicant

Receipt and Review Process at NIH >10,000 applications arrive at given deadline! Receipt/review process organized in 3 cycles per year

Assignment based on Abstract, Specific Aims, more if needed Each application assigned to funding Institute(s)/Center(s) Application also assigned to Integrated Review Group (IRG) within Center for Scientific Review (CSR) or to Institute/Center (IC) review group Assignment Process at NIH

CSR: –R01s, R03s, R21s, Small business, Fellowships –Reviews for >1 Institute IC Review: –Program projects, Training grants, Career development awards, Responses to Requests for Applications –Specific to Institute Assignment Process at NIH

Application then assigned to Study Section NIH officials will consider requests for these assignments –Cover letter Assignment Process at NIH

Process at NIH Scientific Review Administrator (SRA) of Study Section decides on reviewers from within members of Study Section or from ad hoc members

Appointment of Reviewers to Study Section SRA recruits members of Study Section Qualifications –Expert with training and experience in relevant scientific field Level of formal education Quantity and quality of relevant research –PI on research project comparable to those being reviewed

Appointment of Reviewers to Study Section SRA also needs to address –Diversity in gender, race, ethnicity and geographic distribution –Fairness and evenhandedness in review –Willingness to do the work required –Ability to write and present clearly

Types of appointments to study section –Regular: Typically several years –Temporary: One time on standing study section, may lead to regular appointment –Special emphasis panel: One time only Appointment of Reviewers to Study Section

Roster of Study Section Available Online Meeting Roster - ZRG1 MOSS-A 91 (4/6/ /6/2005) CENTER FOR SCIENTIFIC REVIEW SPECIAL EMPHASIS PANEL ZRG1 MOSSA 4/6/2005-4/6/2005 MEETING ROSTER CHAIRPERSON MYERS, ELIZABETH R., PHD ASSOCIATE PROFESSOR WEILL MEDICAL COLLEGE OF CORNELL UNIVERSITY ASSOCIATE SCIENTIST, HOSPITAL FOR SPECIAL SURGERY NEW YORK, NY MEMBERS ADAMS, JOHN S., MD PROFESSOR DEPARTMENT OF MEDICINE CEDARS-SINAI MEDICAL CENTER LOS ANGELES, CA AHLGREN, SARA C., PHD…

SRA matches grant applications to specific reviewers Tries to ensure –Appropriate expertise –Diverse scientific viewpoints Tries to avoid –Overload of particular reviewer –Potential conflict of interest Assignment of Applications to Reviewers

2 reviewers and 1 discussant (typically) are assigned to each proposal –Primary reviewer –Secondary reviewer –Reader (does not need to prepare written review prior to meeting of study section) Could be more – Tertiary, more Readers Assignment of Applications to Reviewers

NIH Review Criteria NIH review criteria for unsolicited research project grant applications (R01, R03, R21) Significance –Important problem –Advancement of scientific knowledge or clinical practice –Influence on methods that drive the field

NIH Review Criteria Approach –Adequate development and integration of design, methods, analyses –Acknowledgment of potential problems, alternatives Innovation –Challenge to existing paradigms –Novel concepts, approaches, methods

NIH Criteria Investigator –Appropriate training, experience –Complementary and integrated team Environment –Conducive to probability of success –Unique features of scientific environment –Institutional support

NIH Criteria Other criteria –Gender/minority/children inclusion –Budget –Protection of humans, animals, and environment Overall rating –Numerical score that reflects overall impact

REVIEW CRITERIA: “K” Awards Candidate Career Development Plan Research Plan Mentor/Co-Mentor(s) Environment & Institutional Commitment

NIH Numerical Rating Priority score: Single, global score for proposal Highest Priority Strong in all categories Average application WORSTBEST Lowest Priority

Guide to Calibrating Score ScoreDescriptors 1.0 – 1.5Outstanding, Close to flawless 1.6 – 2.0Highly significant, Few weaknesses 2.1 – 2.5Weaknesses need to be addressed 2.6 – 3.0Weaknesses balance strengths >3.0Weaknesses outweigh strengths

Submitting Critique Before Meeting Electronic submission of reviews Several days before meeting, reviewers upload score and written critique Once uploaded, can then read other reviewers’ scores and reviews Once uploaded, reviewer cannot make changes to scores or critiques until after meeting

Study Section Meeting Streamlined Applications Definition: –Not in upper half –Priority score higher than 3 Does not apply to career awards, fellowships (R13, R18, F06, F32/33)

Study Section Meeting Streamlining Streamlining Procedure Reviewers asked ahead of time to recommend applications not in upper half (“unscored” or “streamlined”) SRA compiles list List discussed at beginning of meeting Any member may ask for proposal to be discussed

Benefits and rationale Gives time for in-depth discussion of better applications Saves costs if meeting is shortened Reduces work of scientific review administrators Less than 25% of applications will be funded Study Section Meeting Streamlining

If application is streamlined, applicant receives unaltered written critiques Fate of unscored applications? Study Section Meeting Streamlining

Study Section Meeting Review Procedures Review procedure for proposals to be scored –Chair introduces application –Each reviewer gives preliminary numerical score or range –Primary reviewer covers description and comments –Other assigned reviewers add comments

Review procedure, continued –Discussion ensues –Consensus is not necessary –Chair calls for priority rating –Every members scores Study Section Meeting Review Procedures

Resume Summary Statement is prepared –SRA asks reviewers to modify critiques to reflect discussion –SRA writes resume and summary of discussion in front –Summary Statement (“Pink Sheets”) sent to applicant

Final Score Average of all scores multiplied by 100 Example: –Average of raw scores from review panel = 1.88 –Final score = 188

Percentiles Percentiles indicate your rank relative to other applications reviewed by group 0.1 (best) to 99.5 (worst)  percentage of proposals receiving better score during last year Example: Score: 188, Percentile: 11

Applications Used in Percentile Conversion R01 reviewed at standing study section –Percentile of score relative to all scores from current round plus last two rounds (1 year) R01 reviewed at special emphasis panel –May be percentiled against distribution of all CSR scores

Funding Decision Flowchart Funding Decision Flowchart

Thank you