Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archived OER Public Archive Home Page

Improving Peer Review CSR Initiatives PRAC, August 27, 2007 National Institutes of Health U.S. Department of Health and Human Services

 CSR Data  CSR Recent Activities  CSR Organization Initiatives  Realigning CSR Peer Review  Initiatives Under Early Discussion Peer Review at CSR

CSR Data

Number of Applications Submitted to NIH Historical Growth

Changes Impacting CSR Peer Reviews 5 Years Ago Now Applications Received46,00080,000 R01s and R21s Reviewed by CSR21,00039, Years Ago Now Average Number of Applications Reviewed by Reviewers The first problem The second problem

2006: CSR Peer Review Statistics 80,000 applications received 55,000 applications reviewed 18,000 reviewers 238 Scientific Review Administrators 1,800 review meetings

Application Reviewed by CSR

Growth of R01 Applications Reviewed at CSR vs. Other ICs

Growth of R21 Applications Reviewed at CSR vs. Other ICs

CSR Recent Activities

1.Increase Communication and Transparency 2.Increase Uniformity 3.Increase Efficiency 4. Improve Study Section Alignment and Performance Changes in CSR Operations

1. Increase Communication Advisory Committee Division Director IRG Chief SRA 1 SRA 2 Mgmt. Services Cross Cutting Weekly Biweekly Monthly Monthly Monthly Monthly

2. Increase Uniformity Appeal Advisory Committee Best Practices Committee Assessments Appropriate Rosters Conduct of Meetings Structure of Summary Statements Changes in CSR Operations

1.Increase Communication and Transparency 2.Increase Uniformity 3.Increase Efficiency Changes in CSR Operations

Increase Efficiency How we distribute 80,000 applications? Retooled for electronic submission Applications are now submitted electronically Assign applications using text fingerprinting, artificial intelligence software Full Implementation by February 2008

1.Increase Communication and Transparency 2.Increase Uniformity 3.Increase Efficiency 4.Improve Study Section Alignment and Performance Changes in CSR Operations

Biannual IRG Review Schedule Scheduled IRGs Biology of Development and Aging (BDA) Infectious Diseases and Microbiology (IDM) Biobehavioral and Behavioral Processes (BBBP) Cell Biology (CB) Musculoskeletal, Oral and Skin Sciences (MOSS) Oncological Sciences (ONC) Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB) Respiratory Sciences (RES) Renal and Urological Sciences (RUS) Scheduled IRGs Biological Chemistry and Macromolecular Biophysics (BCMB) Cardiovascular Science (CVS) Bioengineering Sciences and Technologies (BST) AIDS and Related Research (AARR) Risk, Prevention, and Health Behavior (RPHB) Genes, Genomes and Genetics (GGG) Digestive Sciences (DIG) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR) Brain Disorders and Clinical Neuroscience (BDCN) Integrative, Functional and Cognitive Neuroscience (IFCN) Molecular, Cellular and Developmental Neuroscience (MDCN) Hematology (HEME) Immunology (IMM) Health of the Population (HOP)

Six Open House Workshops in 2007 Mar. 2,Neurological (3): Brain Disorders and Clinical Neuroscience (BDCN); Integrative, Functional and Cognitive Neuroscience (IFCN); Molecular, Cellular and Developmental Neuroscience (MDCN) April 25Behavioral/Social (3): Biobehavioral and Behavioral Processes (BBBP); Health of the Population (HOP); Risk Prevention and Health Behavior (RPHB) Jun. 29Disease-based (4): AIDS and Related Research (AARR); Infectious Diseases and Microbiology (IDM); Oncological Sciences (ONC); Surgical Sciences, Biomedical Imaging and Bioengineering (SBIB) Aug. 24Integrated Biological (4): Digestive Sciences (DIG); Musculoskeletal, Oral and Skin Sciences (MOSS); Renal and Urological Sciences (RUS) Endocrinology, Metabolism, Nutrition and Reproductive Sciences (EMNR) Nov. 9 Integrated Biological (5): Immunology (IMM); Hematology (HEME); Cardiovascular Sciences (CVS); Respiratory Sciences (RES); Biology of Development and Aging (BDA) Dec. 18Biomolecular (4): Biological Chemistry and Macromolecular Biophysics (BCMB); Bioengineering Sciences and Technologies (BST); Cell Biology (CB); Genes, Genomes and Genetics (GGG)

Neuroscience Open House, March 2007

Neuroscience Open House Workshop Total registrants169 Study Section Chairs26 PRAC Members2 Professional Society Members From 26 different societies 52 NIH staff (non-CSR) 14 ICs represented 33 CSR staff members50 Public members6

CSR Organization Initiatives

CSR Organization

Review Enabling Committees Policy Scientific Community Outreach Recruitment of SRAs Recruitment of Reviewers Knowledge Management Electronic Reviews SRA Training Mentoring and staff development Best review practices

CSR 4 Review Divisions & 23 IRGs Division of Biologic Basis of Disease. AIDS and Related Research (AARR). Endoc, Metabolism, Nutrition, Reproductive Sciences (EMNR) Immunological Sciences (IMM) Oncological Sciences (ONC) Infectious Diseases and Microbiology (IDM). Division of Physiology and Pathology. Cardiovascular Sciences (CVS) Integrative, Functional Cognitive Neuroscience (IFCN). Renal and Urological Sciences (RUS). Hematology (HEME) Digestive Sciences IRG (DIG) Division of Clinical and Population-Based Studies Surgery, Radiology, and Bioengineering (SRB) Health of the Population (HOP) Risk, Prevention, and Health Behavior (RPHB) Brain Disorders and Clinical Neuroscience (BDCN) Behavioral & Biobehavioral Processes (BBBP) Division of Molecular & Cellular Mechanisms. Biology of Development and Aging (BDA) Biological Chemistry and Macromolecular Biophysics (BCMB) Cell Biology (CB). Bioengineering Sciences and Technologies (BST) Genes, Genomes, and Genetics (GGG). Molecular, Cellular Develop Neuroscience (MDCN). Respiratory Sciences (RES) Musculoskeletal, Oral, Skin Sciences (MOSS)

Organization of Peer Review at CSR Divisions Study Sections I.R.G.s (Integrated Review Groups)

New IRG ENST/Emerging Neuroscience & Training

Organization of Peer Review at CSR Divisions Study Sections I.R.G.s (Integrated Review Groups)

New Division Neuroscience, Aging and Development BDCN/Brain Disorders & Clinical Neuroscience (est. 11 SRAs) IFCN/Integrative, Functional, & Cognitive Neuroscience (est. 9 SRAs) MDCN/Molecular, Cellular & Developmental Neuroscience (est. 10 SRAs) ENST/Emerging Neuroscience & Training (est. 8 SRAs) BDA/Biology of Development & Aging (7 SRAs)

Realigning Peer Review

CCSR Recent Activities in Peer Review SR Recent Activities in Peer Review 1.Shorten the review cycle 2.Do more to recruit and retain more high-quality reviewers and decrease the burden on applicants and reviewers 3.Improve the identification of significant, innovative and high-impact research

Shortening the Review Cycle First Step: Posting Summary Statements Earlier Post new investigator within 10 days, (within 30 days for established investigators) Second Step: Pilot to Shorten the Review Cycle for New R01 Investigators Last year: About 2,000 new investigators in pilot Third Step: Shorter Review Cycles to All R01 From New Investigators Feb 2007: 3,000 June 2007: 6,000 new investigators Nov 2007: All new investigators (>9000)

R01 A1 Resubmissions Reviewed in the Cycle Immediately following the Original Application's Review Council Cycle of Original R01 NewExperienced

A Vision for Peer Review 1.Shorten the review cycle 2.Do more to recruit and retain more high-quality reviewers and decrease the burden on applicants and reviewers

CSR’s Growing Need for Reviewers

Reviewers – Current Situation Far too many reviewers on study sections Broader science Decrease in reviewer load Unnecessary Too many ad hoc reviewers

Near-Term Solutions for Recruiting and Retaining the Best Reviewers Remaking peer review a learning experience Massive call for volunteers Rewards for reviewers Require less travel by expanding peer review platforms Shorten applications

CSR Recent Activities in Peer Review

Near-Term Solutions for Recruiting and Retaining the Best Reviewers Require less travel by using electronic review modes

Expanding Peer Review’s Platforms Study Sections Necessity ● Clinical reviewers Preference ● Physicists, computational biologists New Opportunities ● Fogarty, International Reviewers Our Goal: 10% of all reviews to be electronic in 2007 Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions

A Study Section Chair Talks About Asynchronous Electronic Discussions “Finally, in spite of my initial skepticism regarding this “blog-like” review mechanism, all... [reviewers] produced comments, critiques, posed questions in a fashion that I would judge to be the BEST I’ve seen in ~30 years of chairing various review groups.” Craig M. Jackson, Ph.D. President and Principal Scientist Hemosaga Diagnostics Corp.

Near-Term Solutions for Recruiting and Retaining the Best Reviewers Require less travel by using electronic review modes Have Shorter Meetings Shorten Applications

Shorter Applications Trans NIH Committee Communication and Support from Societies, Diseases Groups, Scientific Leadership 2 Retreats of NIH Directors Approval and Support of our Advisory Committee Survey of Scientists

Initiatives Under Early Discussion

Initiatives Under Early Planning/Consideration 12 Pilots with short applications Continuous receipt of applications for chartered study section members Editorial board reviews

Deeper Changes? The NIH ACD Subcommittees Locus of review NIH Support for Investigator The Role of Council NIH Portfolio Management Firewall between review and program Scoring Review of deeply innovative, translational and multidisciplinary research Others?

This is CSR

Growth of R01 Applications Reviewed at CSR vs. Other ICs

Growth of R21 Applications Reviewed at CSR vs. Other ICs

Major Factors Affecting Payline 1. NIH Budget 2. NIH Budget 3. NIH Budget 4. NIH Budget

Major Factors Affecting Payline 1. Number of Application Submitted 2. Targeting of the Budget 3. Recycling of Funds 4. NIH Budget

CSR Review of Women Applications, Women Scoring 10 and 20 %

NIH 20%tile or Better Scoring (R01s, CSR reviews) CouncilsNew Applications New Investigators (Type 1) New Applications Established Investigators (Type 1) Renewals From Established Investigators (Type 2) May %17%34% May %17%34% May %18%33% May %18%34% May %18%33% CSR New and Established Investigators Score%

M. Martin, CSR/NIH/DHHS

Significant Numbers of Clinical Grantees Are Not Submitting Renewal Applications R01 Applications HS+HS- Type 1 New A024.59%20.20% Type 2 A011.71%17.11% M. Martin, CSR/NIH/DHHS

CSR Budget

CSR Budget for 2007 CSR Budget $ 60 M SREA Budget $ 40 M Review Cost, including travel for 18,000 reviewers, is less than 0.2% of the budget requested in the applications reviewed

Travel Purchase non-refundable (restricted) coach tickets instead of unrestricted tickets Saving – June 2006 through July 2007 # of tickets issued 28,493 Changes requested 2544 cost of changes + fees $550,274 YTD Fare Savings$14,522,877