Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change NIDA Council Bethesda, Feb 8, 2006

Peer Review: An N.I.H. “Conception” Is the heart and soul of NIH Has produced an effective partnership between the federal government and research institutions Has created the best academic medical centers, the best biomedical/behavioral research and biotechnology Has made possible the best cures and the best prevention Has been admired and imitated here and abroad Has protected NIH against outside influence

Center for Scientific Review

This Is Not Amazon.com This is CSR

Applications Received for all of NIH and Applications Referred for CSR Review, FY NIH Applications Applications for CSR Review

CSR Mission Statement To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.

NIDA Applications Reviewed by CSR in FY 2002 and Number of applications reviewed by CSR (for all ICs) 33,46951,690 Number of NIDA applications reviewed by CSR Percent of total reviewed by CSR 969 (2.9%) 1,480 (2.9%) Number of CSR study section meetings 1,3871,798 Number of CSR study sections meetings that reviewed NIDA applications (Percent of total) 225 (16.2%) % Fiscal Year of Review

Review Outcomes for NIDA Applications in 2002 and 2005 R01 Applications Only FYN % Unscored Median Priority Scores % Scoring at the 10 th percentile or better % Scoring at the 20 th percentile or better % %18.6% %* % 20.5% * FY comparison, p <.001

Time Complexity and Impact CSR Operations Current Systems New Systems? Necessary Changes in CSR Peer Review Operations

Increase communications between CSR, the ICs, our reviewers and applicants Increase uniformity Increase efficiency Facilitate work of IC program staff Changes in CSR Operations

Increased Communication and Transparency Within CSR With NIH and other Agencies With the Scientific Community Changes in CSR Operations 1

Increase uniformity Slate Nomination Summary Statements Posting all within one month of Study Section meetings Posting Summary Statements of new investigators within one week Producing more complete and structured resumes Unscoring Common practice Unscoring 50% Changes in CSR Operations 2

Increase Efficiency Electronic Submission Text Fingerprinting, Artificial Intelligence Software Changes in CSR Operations 3

Potential of Knowledge Management Tools for Peer Review Collexis Software or Others Knowledge management solutions Fingerprinting and text retrieving Disease coding Benefits for Peer Review Assigning applications to Integrated Review Groups or Study Sections Selecting reviewers (one application, multiple applications) Nine pilots are underway to begin to assess these benefits

Facilitate work of IC program staff Possible Changes in CSR Operations

Study Section Realignment Review of one IRG every month Total review every 2 years

Required Changes in Current Systems Shorten the review cycle

This is Not an Ford Assembly Line ReceiptRefer Evaluate Scientific Merit of Applications

Shortening the NIH Review Cycle, Initial Steps For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05) We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)

Possible Changes in Current Systems Shorten the review cycle Address concern that clinical research is not properly evaluated Improve the assessment of innovative, high- risk/high-reward research Do more to recruit and retain more high-quality reviewers

Expanding Peer Review’s Platforms Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions Study Sections Necessity ● Clinical reviewers Preference ● Physicists, computational biologists New Opportunities ● Fogarty, International Reviewers

Applications Received for All of NIH FY ,000 40,000 60,000 80, Fiscal year Number of applications

Number of Research Grant Applications/Applicant

CSR Applications Reviewed, Regular and SEP May Council Only

Study Section Application/Reviewer Ratio October Council Only

CENTER FOR SCIENTIFIC REVIEW FY Non-Discretionary vs. Discretionary Spending ■ Non-Discretionary ■ Discretionary FY 2004FY 2005 FY 2006

If we didn’t have any peer-review system and we had to design one from scratch, what would it look like? Possible New Systems

This is CSR