NIH Peer Review: Continuity and Change Toni Scarpa Center for Scientific Review National Institutes of Health Department of Health and Human Services APS/Cardiovascular Section Salt Francisco, April 2006

Peer Review: An N.I.H. “Conception” Is the heart and soul of NIH Has produced an effective partnership between the federal government and research institutions Has created the best academic medical centers, the best biomedical/behavioral research and biotechnology Has made possible the best cures and the best prevention Has been admired and imitated here and abroad Has protected NIH against outside influence

Center for Scientific Review

Applications received for all of NIH and applications referred for CSR review, FY 1998-2004 80,000 60,000 Number of applications 40,000 20,000 1998 2000 2002 2004 Fiscal year Applications received for all of NIH Applications assigned for review by CSR

CSR 4 Review Divisions & 23 IRGs Division of Biologic Basis of Disease Elliot Postow, Ph.D. Division of Molecular and Cellular Mechanisms Donald Schneider, Ph.D. Division of Physiology and Pathology Michael Martin, Ph.D. Division of Clinical and Population-Based Studies Anita Miller Sostek, Ph.D AIDS and Related Research IRG (AARR) Ranga V. Srinivas, Ph.D. Bioengineering Sciences and Technologies IRG (BST) Sally Amero, Ph.D. Cardiovascular Sciences IRG (CVS) Joyce Gibson, D.Sc. Behavioral & Biobehavioral Processes IRG (BBBP) Karen Sirocco, Ph.D. Biology of Development and and Aging (BDA) Sherry Dupere, Ph.D. Digestive Sciences IRG (DIG) Mushtaq Khan, Ph.D., DVM Brain Disorders and Clinical Neuroscience IRG (BDCN) Dana Plude, Ph.D. Endocrinology, Metabolism, Nutrition, and Reproductive Sciences (EMNR) Sooja Kim, Ph.D. Biological Chemistry and Macromolecular Biophysics IRG (BCMB) John Bowers, Ph.D. Hematology IRG (HEME) Joyce Gibson, D.Sc. Health of the Population IRG (HOP) Robert Weller, Ph.D. Immunological Sciences IRG (IMM) Calbert Laing, Ph.D. Integrative, Functional and Cognitive Neuroscience IRG (IFCN) Christine Melchior, Ph.D. Cell Biology IRG (CB) Marcia Steinberg, Ph.D. Risk, Prevention, and Health Behavior IRG (RPHB) Michael Micklin, Ph.D. Infectious Diseases and Microbiology IRG (IDM) Alex Politis, Ph.D. Musculoskeletal, Oral, and Skin Sciences IRG (MOSS) Daniel McDonald, Ph.D. Genes, Genomes, and Genetics IRG (GGG) Richard Panniers, Ph.D. Surgery, Radiology, and Bioengineering IRG (SRB) Eileen Bradley, D.Sc. Oncological Sciences IRG (ONC) Syed Quadri, Ph.D. Renal and Urological Sciences IRG (RUS) Daniel McDonald, Ph.D. Molecular, Cellular and Developmental Neuroscience IRG (MDCN) Carole Jelsema, Ph.D. Respiratory Sciences IRG (RES) Mushtaq Khan, Ph.D., DVM

CSR Mission Statement To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.

Necessary Changes in CSR Peer Review Operations CSR Operations Current Systems New Systems? Complexity and Impact Time

Changes in CSR Operations Increase communications between CSR, the ICs, our reviewers and applicants Increase uniformity Increase efficiency Facilitate work of IC program staff

Potential of Knowledge Management Tools for Peer Review Collexis Software or Others Knowledge management solutions Fingerprinting and text retrieving Disease coding Benefits for Peer Review Assigning applications to Integrated Review Groups or Study Sections Selecting reviewers (one application, multiple applications) Nine pilots are underway to begin to assess these benefits

Study Section Realignment Review of one IRG every month Total review every 2 years

Required Changes in Current Systems Shorten the review cycle

This is Not an Ford Assembly Line Evaluate Scientific Merit of Applications Receipt Refer EnterpriseArchitecture@mail.nih.gov

Shortening the NIH Review Cycle, Initial Steps For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05) We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)

Desirable Changes in CSR Review Shorten the review cycle Address concern that clinical research is not properly evaluated Improve the assessment of innovative, high- risk/high-reward research

“The judging of grants has become a charade.” The American Society for Cell Biology “The judging of grants has become a charade. To be funded, the experimental plan has become a litany of experiments already accomplished so that everything is feasible. When grants come back with unfundable scores, new investigators may not have sufficient resources to do the experiments that “show feasibility.” Zena Werb President, ASCB Newsletter August 2005

Possible Changes in Current Systems Shorten the review cycle Address concern that clinical research is not properly evaluated Improve the assessment of innovative, high- risk/high-reward research Do more to recruit and retain more high-quality reviewers

Expanding Peer Review’s Platforms Study Sections Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions Necessity ● Clinical reviewers Preference ● Physicists, computational biologists New Opportunities ● Fogarty, International Reviewers

The First NIH Study Section 1945 The Last NIH Study Section 2005

Possible New Systems If we didn’t have any peer-review system and we had to design one from scratch, what would it look like?

Questions—Applications Should we have 3 or 365 deadlines for most applications? Should applications (Rs) be shortened? Should appendices be eliminated or reduced in size? Is there more value in having 2-3 reviewers reading 25-page applications or 10-15 reviewers reading 5-page applications?

Questions—Study Section Meetings What is the ideal number of members to have serve on a study section? Is one study section with 50-70 reviewers efficient? What is the intellectual contribution of individual reviewers in large study sections?

Questions—Scoring Is it proper or desirable to have 50-70 reviewers voting on priority scores for each application referred to their study section? Is consensus always good? Or should we focus on score variance?

Applications Received for All of NIH FY 1998-2004 80,000 60,000 Number of applications 40,000 20,000 1998 2000 2002 2004 Fiscal year

Number of Research Grant Applications/Applicant

CSR Applications Reviewed, Regular and SEP May Council Only

Study Section Application/Reviewer Ratio October Council Only

Two groups of challenges/opportunities Reviewers Decrease the number of reviewers and increase the quality Increase the number of applications reviewed without extra workload Recruit and retain the best reviewers SRA Increase efficiency Recruit and train

Possible Short Term Approaches for Increasing Efficiency for Reviewers and CSR Replace Many SEPs with Smaller Parallel Study Sections Enlarge Study Section Membership and Decrease Frequency of Participation Pre Meeting Streamlining Various Review Platforms Hybrid Review Platforms Staggering Application Deadlines 2 instead of 3 reviews Shorten Applications More Structured Applications and Reviews

Size of Grant Applications RO1 Will increase number of applications reviewed by reviewers Will decrease the number of reviewers in a study section May be combined with a change in format of the application, more consonant to review criteria May be combined with scoring individual criteria (i.e. relevance, innovation, etc Strong support by councils and scientific leadership May result in more innovation

This is CSR

Coronary Heart Disease Age-Adjusted Death Rates in U.S.: Actual (blue) vs Expected (yellow) 500 400 300 200 100 1950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000 Deaths per 100,000 Year ~ 514,000 Actual Deaths in 2000 ~ 1,329,000 Projected Deaths 815,000 Deaths Prevented in 2000

The True Value of Peer Review