Challenges and Opportunities in Peer Review A Vision for Ensuring Its Strategic National Value toni scarpa FASEB Board Bethesda, MD May 31, 2009 National Institutes of Health U.S. Department of Health and Human Services

NIH and Peer Review The Drivers for Change Enhancing Peer review: CSR Enhancing Peer review: Corporate NIH ARRA, The Stimulus NIH Peer Review

30 Years of Medical Innovation MRI and CT Imaging ACE inhibitors Angioplasty Statins Mammography Coronary Interventions H inhibitors and H2 Blockers Antidepressant Cataract and Lens Replacement Knee and Hip Replacement Ultrasound Imaging Asthma Treatment Cardiac Enzymes Fluoroquinolones Hypoglycemic Agents HIV Testing and Intervention Tamoxifen PSA H. Pylori Test and Treatment Bone Densitometry Cephalosporins Calcium Blockers Conscious Sedation Fuchs and Sox, Health Affairs, 20, 30-42

Why Has The U.S. Biomedical-Behavioral Research Been So Successful? Evolution of unique dynamic partnerships -- through NIH -- between Government and academic/medical schools 100% of NIH funds to universities and medical centers awarded through peer review (Only 5-20% in Europe)

The Overarching Influence of NIH Peer Review in the USA For U.S. Universities and Academic Medical Centers The promotion committee of medical schools The prestige and finances of universities and medical schools For People and for Public Health Which research is done Which cures people get 7000 Diseases Affect Humankind 6000 are Orphan Diseases

CSR Peer Review: ,000 applications received 16,000 reviewers 1,600 review meetings 240 Scientific Review Officers 2,500

CSR Peer Review: ,000 applications received 16,000 reviewers 1,600 review meetings 240 Scientific Review Officers 115,000 38,000 2,500

The Drivers for Change

$13.7 $15.6 $17.8 $20.5 $23.3 $27.1 $28.0 $28.6 $29.1 $29.5 $0 $5 $10 $15 $20 $25 $ st Driver: The NIH Budget Doubling $ In Billions

2 nd Driver: Number of Applications Historical Growth

3 rd Driver: Reviewers Load Applications Per Reviewer

4th Driver: CSR Budget $ Millions

Annual Savings in Reviewers Expense Budget Non-refundable tickets with one possible change $15 million 3,000 fewer reviewers $3 million 15% reviews using electronic platforms $5 million One meeting a year on the West Coast $1.8 million

CSRs Efforts to Enhance Peer Review

Major Complaints About NIH Peer Review The process is too slow There are not enough senior/experienced reviewers The process favors predictable research instead of significant, innovative, or transformative research The time and effort required to write and review are a heavy burden on applicants and reviewers

1.Reorganizing CSR and Recruiting Staff 2.Improving Study Section Alignment 3. Shortening the Review Cycle 4. Advancing Additional Review Platforms 5. Recruiting the Best Reviewers 6.AATS Peer Review Outcome CSRs Efforts to Enhance Peer Review

1. Reorganizing CSR Translational and Clinical Sci Cardiovascular and Respiratory Sciences Surgical Sciences, Biomedical Imaging and Bioengineering Musculoskeletal, Oral And Skin Sciences Oncology 2 – Translational Clinical Vascular and Hematology Physiological Pathological Sci Endocrinology, Metabolism, Nutrition & Reproductive Sciences Immunology Infectious Diseases & Microbiology Digestive, Kidney & Urological Systems Neuroscience, Development and Aging Brain Disorders & Clinical Neuroscience Molecular, Cellular & Developmental Neuroscience Integrative, Functional & Cognitive Neuroscience Emerging Technologies & Training in Neuroscience Biology of Development & Aging Biobehavioral & Behavioral Processes Risk, Prevention& Health Behavior Population Sciences and Epidemiology Healthcare Delivery & Methodologies AIDS & Related Research AIDS, Behavioral, Population Basic- Integrative Biological Sci Biological Chemistry & Macromolecular Biophysics Bioengineering Sciences & Technologies Genes, Genomes &Genetics Oncology 1 – Basic Translational Cell Biology Interdisciplinary Molecular & Training

2. Improving Study Section Alignment Input from the community Internal IRG reviews Open Houses PRAC

3. Shortening the Review Cycle Why? First Response was 5.2 months, far too long The Goal To review and post score and critique application within 3 months of submission. To enable resubmission, when doable and desirable, 4 months earlier than in the past. The Result Every New Investigator and Most Established Investigators are eligible 13% of those eligible apply

4. Advancing Additional Review Platforms Additional Review Platforms Help Recruiting Reviewers Electronic Review Modes Reduce Travel Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions (AED)

4. Advancing Additional Review Platforms Reviewer Satisfaction with AED Review

5. Recruiting the Best Reviewers

5. Recruiting the Best Reviewers Academic Rank of ALL CSR Reviewers

5. Recruiting the Best Reviewers Some Successful Strategies Move a meeting a year to the West Coast Additional review platforms Develop a national registry of volunteer reviewers Searchable database with 5,000 reviewers Provide tangible rewards for reviewers No submission deadlines for chartered members of study sections (effective February 2008). Provide flexible time for reviewers Choice of 3 times/year for 4 years or 2 times/year for 6 years

Enhancing Peer Review The NIH Directors Recommendations

Corporate NIH: Enhancing Peer Review The Charge from Dr. Zerhouni: Fund the best science, by the best scientists, with the least administrative burden… Two advisory committees to the NIH Director

The Process Diagnostic Design Implementatio n Plan Begin Phased Implementatio n of Selected Actions June 2007 – Feb March 2008 – June 2008September

1. Early Stage Career Investigators : Definition of New Investigator: Not previously competed successfully as PD/PI for a significant NIH independent research award. Definition of Early Stage Investigator: Within 10 years of completing terminal research degree or within 10 years of completing medical residency (or the equivalent). The NIH corporate policy is to fund R01s of New Investigators and ESIs at different paylines 3 Paylines for R01s Applies only to R01 applications Advice for ESI (and to New PI) Apply NOW!!! Apply for R01!!!!!

2. Review Highly Transformative Research OD Transformative RO1 (T-RO1) Once a year, 5 years, $40 million total budget Deadline January 29, page application 740 Editorial Board Review oHeavy triage based on innovation and potential science transformation by a small study section of distinguished, broad-science reviewers (the editors) oSpecific science reviewed by appropriate reviewers (the editorial board) oFinal ranking by the editors

3. Funding the Best Research Earlier More flexible deadlines Abolish A2 applications

4. Enhanced Review Overall Impact: ) Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) New Core Criteria Order: Significance Investigator(s) Innovation Approach Environment

4. Template-Based Critiques The objective is to write evaluative statements and to avoid summarizing the application Comments should be in the form of bullet points or if necessary short narratives The entire template is uploaded to IAR to become part of the summary statement. Significance 1. Significance Please limit text to ¼ page Strengths Weaknesses

4. Scoring ImpactScoreDescriptor High Impact 1Exceptional 2Outstanding 3Excellent Moderate Impact 4Very Good 5Good 6Satisfactory Low Impact 7Fair 8Marginal 9Poor

4. The Essence of Scoring Before the Meeting Every assigned reviewer will post criteria scores AND overall impact scores. The impact scores will be used to determine the order of discussion. During the Meeting For the discussed applications: The overall impact score is stated by the assigned reviewers. Criteria scores are not mentioned during the discussion. After the Meeting The applicant with a discussed application will see 3 or more criteria scores AND the overall impact score (the one that will be percentiled) The applicants with a non discussed application will see ONLY the criteria scores (3 or more for each criterion)

4. Enhancing Peer Review Training CSR and NIH Review Staff 6 face to face training sessions, January face to face training sessions, April 2009 Continuous updating Chairs For Chairs appointed in 2008, 8 sessions in January 2009 For Chairs appointed in 2009, 7 sessions in April-May 2009 Reviewers Training material (Power Point, interactive training, frequently asked questions, mock study section video, etc,in April-May 2009 Senior CSR staff at the first meeting in May-July 2009

American Recovery and Reconstruction Act

The Stimulus

ARRA Applications Under Review RC1. Challenge Grants Deadline April 27 Received so far 20,847 Verified and Assigned to IRGs19,107 Assigned to 3 Reviewers14,313 2 Stages Editorial Board Review oEach application assigned to 3 reviewers (20,000), reviews due by June 5 th, extended to June 12th o30 Special Study Sections in early July Competitive Revisions Deadline April 23 Received 1,985 Large Majority Assigned to the Standard 250 Study Sections Study Sections will review as a SEP in May-June Stem Cells Challenges

This is CSR September 2008