Understanding NIH Peer Review David B. Winter, Ph.D. Scientific Review Officer Cellular and Molecular Immunology-A (CMIA) Study Section Center for Scientific Review, NIH

NIH – 27 Institutes and Centers Office of the Director National Library of Medicine Center for Information Technology Scientific Review Fogarty International Center National Institute of Arthritis and Musculoskeletal and Skin Diseases of Diabetes and Digestive and Kidney Diseases of Dental and Craniofacial Research National Institute on Deafness and Other Communication Disorders National Eye Institute National Heart, Lung, and Blood on Alcohol Abuse and Alcoholism National Cancer on Drug Abuse of Environmental Health Sciences of Mental Health of Neurological Disorders and Stroke of Allergy and Infectious Diseases National Center on Minority Health and Health Disparities on Aging of Child Health and Human Development National Human Genome Research of General Medical Sciences of Nursing Research National Center for Complementary and Alternative Medicine for Research Resources Clinical Center National Institute of Biomedical Imaging and Bioengineering no funding authority Component of the Department of Health and Human Services (DHHS) 27 Institutes and Centers and the Office of the Director (Francis Collins M.D., Ph.D.) over 18,000 Employees Headquartered in Bethesda, MD. Off-campus sites in Rockville and Bethesda, MD; North Carolina; Michigan; Montana; and Arizona National Institute on Minority Health and Health Disparities Different Missions, Responsibilities and Constituencies

NIH Extramural Staff Program Officer (PO) Scientific Review Officer (SRO) Grants Management Specialist (GMS)

Program Officer A science professional, usually Ph.D./M.D. level, who: Serves as an advocate for investigators Provides scientific stewardship and administration of grants & contracts awarded by NIH Identifies areas of scientific priority and develops funding opportunities for extramural researchers Provides guidance on NIH extramural policy/procedures, research resources, and funding opportunities to extramural investigators

Scientific Review Officer A science professional, usually M.D. or Ph.D. level, who: Is based at an NIH Institute or Center (IC), or at the Center for Scientific Review (CSR) Manages Study Sections and review panels for grants and contracts Selects review panel members Assigns reviewers to applications Compiles application summary statements

Grants Management Specialist Business/finance professional who: Negotiates and awards all grants Provides fiscal administration of grants Is the government official on fiscal and policy issues and approvals

NIH Peer Review System for Grant Applications First Level of Review Scientific Review Group (Study Section) Second Level of Review NIH Institute/Center Advisory Council

CSR assigns applications to study section (review) & NIH Institute (funding) Researcher writes and Institution submits application to NIH Summary Statement and Priority Score transmitted to applicant (Commons) and NIH extramural staff Study section composed of 20-30 reviewers – review/discuss applications 3- 4 reviewers assigned to read and write critiques for each application

Center for Scientific Review Division of Receipt and Referral Central receipt point for most PHS Grant Applications Institute/Center assignment (potential funding component) Assignment to Scientific Review Group in CSR or in an Institute Scientific Review Groups Approximately 240 CSR chartered study sections and regularly recurring Special Emphasis Panels that primarily review: Research Grant Applications Fellowship Applications Academic Research Enhancement Award Applications Small Business Innovation Research Applications

CSR Web Site http://www.csr.nih.gov About CSR Applicant Resources Study Sections Rosters and Meetings http://www.csr.nih.gov

Division of Physiological and Pathological Sciences Integrated Review Groups (IRGs) Digestive, Kidney, and Urological Systems Endocrinology, Metabolism, Nutrition, and Reproductive Sciences Immunology (IMM ) Infectious Diseases and Microbiology Cellular and Molecular Immunology A Cellular and Molecular Immunology B Hypersensitivity, Autoimmune, and Immune-Mediated Diseases Immunity and Host Defense Innate Immunity and Inflammation Transplantation, Tumor, and Tolerance Immunology Vaccines Against Microbial Diseases

Cover Letter ONLY NIH STAFF HAVE ACCESS TO YOUR COVER LETTER! The cover letter should be used for a number of important purposes: Suggest Institute/Center assignment Suggest review assignment Identify individuals in potential conflict and explain why Identify areas of expertise needed to evaluate the application Discuss any special situations It is NOT appropriate to use the cover letter to suggest specific reviewers.

How Reviewers Are Selected for Study Section Service Demonstrated scientific expertise/research support Doctoral degree or equivalent Mature judgment Work effectively in a group context Breadth of perspective Impartiality Representation of women and minority scientists Geographic distribution

Integrity of the NIH Peer Review Process Confidentiality Review materials and proceedings of review meetings represent privileged information for reviewers and NIH staff. Applicants should never communicate directly with any members of the study section about an application. Conflicts of Interest (COI) Institutional, family member/close friend, collaborator, longstanding scientific disagreement, personal bias, appearance of conflict http://grants.nih.gov/grants/peer/peer_coi.htm

Before the Study Section Meeting Each application is assigned to 3 or more reviewers 5-6 weeks in advance Reviewers assess each application by providing: A preliminary Overall Impact score Criterion Scores for each of the 5 Core Review Criteria A written critique

What Reviewers Look for in Applications Significance and impact Exciting ideas Clarity Ideas they can understand -- Don’t assume too much Realistic aims and timelines -- Don’t be overly ambitious Brevity with things that everybody knows Noted limitations of the study A clean, well-written application In thinking upon what one of your peers might generally look for as a reviewer of an application keep the following practical ideas in mind:

Common Problems in Applications Lack of new or original ideas Lack of a clearly stated hypothesis Absence of an acceptable scientific rationale Lack of experience in the essential methodology Questionable reasoning in experimental approach Uncritical approach Diffuse, superficial, or unfocused research plan Lack of sufficient experimental detail Lack of knowledge of published relevant work Unrealistically large amount of work Uncertainty concerning future directions

Review criteria each scored from 1-9 Overall Impact Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved Core Review Criteria Significance Investigator(s) Innovation Approach Environment Review criteria each scored from 1-9

Significance vs. Overall Impact The Significance criterion – Assumes success Assuming that all the aims are successful, does the project address a problem or critical barrier to progress in the field or have the ability to improve knowledge, technical capability, or clinical practice in a major (1-3), moderate (4-6) or minor (7-9) way? Overall Impact – can be influenced by all 5 criteria (significance, investigator, innovation, approach, environment) weighted based on reviewer’s judgment The high (1-3), medium (4-6) or low (7-9) likelihood that a project will have a sustained and powerful influence on the science.

Additional Criteria Contribute to Overall Impact Scores Protections for human subjects Inclusions of women, minorities, and children Appropriate use of vertebrate animals Management of biohazards

Scoring 9-point score scale is used to provide: Criterion Scores for each of the 5 core review criteria Overall Impact/Priority Score based on but not a sum of the core criterion scores plus additional criteria All applications receive scores: Not discussed applications will receive only initial criterion scores from the three assigned reviewers. Discussed applications also receive an averaged overall impact score from eligible (i.e., without conflicts of interest) panel members.

The Study Section Meeting

CSR Study Sections: The Meeting Each CSR standing Study Section has ~12-22 regular members plus temporary reviewers from the scientific community About 70-90 applications are usually reviewed by each study section in 1-2 day meetings

At the Meeting Order of Review The average of the preliminary Overall Impact score from the assigned reviewers determines the review order Discussions start with the application with the best average preliminary Overall Impact score Clustering of Review New Investigator R01 applications are clustered Clinical applications & other mechanisms may be clustered (n ≥ 10) Not Discussed Applications About half the applications will be discussed Applications unanimously judged by the review committee to be in the lower half are not discussed

Summary Statement Program Officer Impact/Priority Score 10-90 range Percentile in whole numbers Percentile: 29

Your Application Was Reviewed What Do You Do Next? Visit NIH’s Next Steps Website http://grants.nih.gov/grants/next_steps.htm

Who Can Answer Your Questions? Before You Submit Your Application A Program Officer at an NIH Institute or Center Scientific Review Officer After You Submit Your Scientific Review Officer After Your Review Your Assigned Program Officer Grants Info: GrantsInfo@nih.gov – 301 435-0714

NIH and CSR Information Sources

NIH Guide For Grants and Contracts U.S. Department of Health and Human Services Announces NIH Scientific Initiatives Provides NIH Policy and Administrative Information Supplies links to application forms Available on the NIH Web Site: http://www.nih.gov NIH Grant Writing Tips: http://grants.nih.gov/grants/grant_tips.htm

NIH Peer Review Information on the Web National Institutes of Health: http://www.nih.gov Office of Extramural Research http://www.nih.gov/grants/oer.htm Grants Policy http://www.nih.gov/grants/policy/policy.htm Electronic Submission http://era.nih.gov/ElectronicReceipt Comprehensive Funding Information for NIH Grants and Contracts http://ProjectRePORTER.NIH.gov Center for Scientific Review: http://www.csr.nih.gov Resources for Applicants http://www.csr.nih.gov/ResourcesforApplicants CSR Study Section Descriptions http://public.csr.nih.gov/StudySections CSR Rosters and Meeting Dates http://public.csr.nih.gov/RosterAndMeetings CSR Peer Review Notes http://www.csr.nih.gov/publications/