Download presentation
Presentation is loading. Please wait.
Published byOsborn Briggs Modified over 9 years ago
1
Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page
2
Challenges and Opportunities in Peer Review A Vision for Ensuring Its Strategic National Value toni scarpa scarpat@csr.nih.gov 301-435-1109 Peer Review Advisory Committee Rockville, MD June 8, 2009 National Institutes of Health U.S. Department of Health and Human Services
3
Peer Review at CSR CSR News Enhancing Peer review Review of ARRA Applications Enhancing Peer Review
4
Peer Review at CSR Enhancing Peer Review
5
$13.7 $15.6 $17.8 $20.5 $23.3 $27.1 $28.0 $28.6 $29.1 $29.5 $0 $5 $10 $15 $20 $25 $30 1998 199920002001 2002 2003 2004 20052006 2007 2008 2009 The NIH Budget Doubling $ In Billions
6
Number of Applications Historical Growth
7
CSR Peer Review: 2008 77,000 applications received 16,000 reviewers 1,400 review meetings 240 Scientific Review Officers
8
CSR Peer Review: 2009 77,000 applications received 16,000 reviewers 1,400 review meetings 240 Scientific Review Officers 115,000 38,000 1,600
9
Peer Review at CSR CSR News Enhancing Peer Review
10
1.Reorganizing CSR and Recruiting Staff 2.Improving Study Section Alignment 3.Assigning Applications more Accurately 4. Shortening the Review Cycle 5. Advancing Additional Review Platforms 6.Special Emphasis Panels 7. Recruiting the Best Reviewers CSR News
11
1. Reorganizing CSR Translational and Clinical Sci Cardiovascular and Respiratory Sciences Surgical Sciences, Biomedical Imaging and Bioengineering Musculoskeletal, Oral And Skin Sciences Oncology 2 – Translational Clinical Vascular and Hematology Physiological Pathological Sci Endocrinology, Metabolism, Nutrition & Reproductive Sciences Immunology Infectious Diseases & Microbiology Digestive, Kidney & Urological Systems Neuroscience, Development and Aging Brain Disorders & Clinical Neuroscience Molecular, Cellular & Developmental Neuroscience Integrative, Functional & Cognitive Neuroscience Emerging Technologies & Training in Neuroscience Biology of Development & Aging Biobehavioral & Behavioral Processes Risk, Prevention& Health Behavior Population Sciences and Epidemiology Healthcare Delivery & Methodologies AIDS & Related Research AIDS, Behavioral, Population Basic- Integrative Biological Sci Biological Chemistry & Macromolecular Biophysics Bioengineering Sciences & Technologies Genes, Genomes &Genetics Oncology 1 – Basic Translational Cell Biology Interdisciplinary Molecular & Training
12
Three New CSR Division Directors Sy Garte Joy Gibson René Etcheberrigaray Division of Translational & Clinical Sciences Division of Physiological & Pathological Sciences Division of Neuroscience, Development and Aging
13
1. Reorganizing CSR Translational and Clinical Sci Cardiovascular and Respiratory Sciences Surgical Sciences, Biomedical Imaging and Bioengineering Musculoskeletal, Oral And Skin Sciences Oncology 2 – Translational Clinical Vascular and Hematology Physiological Pathological Sci Endocrinology, Metabolism, Nutrition & Reproductive Sciences Immunology Infectious Diseases & Microbiology Digestive, Kidney & Urological Systems Neuroscience, Development and Aging Brain Disorders & Clinical Neuroscience Molecular, Cellular & Developmental Neuroscience Integrative, Functional & Cognitive Neuroscience Emerging Technologies & Training in Neuroscience Biology of Development & Aging Biobehavioral & Behavioral Processes Risk, Prevention& Health Behavior Population Sciences and Epidemiology Healthcare Delivery & Methodologies AIDS & Related Research AIDS, Behavioral, Population Basic- Integrative Biological Sci Biological Chemistry & Macromolecular Biophysics Bioengineering Sciences & Technologies Genes, Genomes &Genetics Oncology 1 – Basic Translational Cell Biology Interdisciplinary Molecular & Training
14
2. Improving Study Section Alignment Don Schneider
15
3. Assigning Applications Accurately
16
4. Shortening the Review Cycle Why? First Response was 5.2 months, far too long The Goal To review and post score and critique application within 3 months of submission. To enable resubmission, when doable and desirable, 4 months earlier than in the past. The Result Every New Investigator and Most Established Investigators are eligible 13% of those eligible apply
17
5. Special Emphasis Panels
18
15% of Applications Percentiling Rosters
19
6. Advancing Additional Review Platforms Additional Review Platforms Help Recruiting Reviewers Electronic Review Modes Reduce Travel Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions (AED)
20
6. Advancing Additional Review Platforms Reviewer Satisfaction with AED Review
21
6. Advancing Additional Review Platforms Cost Comparison of Review Platforms PhoneAEDVEDFace to Face Cost/application25107237867 Cost/reviewer311002921767 Not including honoraria for reviewers
22
7. Recruiting the Best Reviewers Some Successful Strategies Moving a meeting a year to the West Coast Additional review platforms Develop a national registry of volunteer reviewers Searchable database with 5,000 reviewers Provide tangible rewards for reviewers No submission deadlines for chartered members of study sections (effective February 2008). Provide flexible time for reviewers Choice of 3 times/year for 4 years or 2 times/year for 6 years
23
7. Recruiting the Best Reviewers Expansion of No Submission Deadlines Present (since February 2008) Chartered Study Section Members oCSR 3127 oOther ICs10124,139 Planned for 2009 Frequent Reviewers* 1323 BSC regular members 260 NAC members 3931, 976 * 6 meetings/last 18 months
24
Enhancing Peer Review
25
The Process Diagnostic Design Implementatio n Plan Begin Phased Implementatio n of Selected Actions June 2007 – Feb. 2008 March 2008 – June 2008September 2008 2
26
Summary of Recommendations Priority 1. Engage the Best Reviewers Priority 2. Improve Quality and Transparency of Review Priority 3. Ensure Balance Across Scientific Fields and Career Stages Priority 4. Continuous Review of Peer Review
27
Enhancing Peer Review 1. Review Highly Transformative Research 2. Fund Early Stage Career Investigators 3. Fund the Best Research Earlier and Reduce the Burden 4. Improve Quality and Transparency of Peer Review 5. Recruit and Retain the Best Reviewers 6. Train Chairs and Reviewers
28
1. Review Highly Transformative Research OD Transformative RO1 (T-RO1) Once a year, 5 years, $20-$40 million total budget Deadline January 29, 2009 8-page application 740 applications Editorial Board Review oHeavy triage based on innovation and potential science transformation by a small study section of distinguished, broad-science reviewers (the editors) oSpecific science reviewed by appropriate reviewers (the editorial board) oFinal ranking by the editors (Face to face, June 17,2009)
29
0% 1% 2% 3% 4% 5% 6% 7% PIs in 1980 Percent of PIs 2. Early Stage Career Investigators Projection of Age Distribution of NIH RPG Investigators: 2020 2530354045505560657075808590 Age
30
2. Early Stage Career Investigators (ESI) : Definition of New Investigator: Not previously competed successfully as PD/PI for a significant NIH independent research award. Definition of Early Stage Investigator: Within 10 years of completing terminal research degree or within 10 years of completing medical residency (or the equivalent). The NIH corporate policy: to fund R01s of New Investigators and ESIs at different paylines. 3 Paylines for R01s Applies only to R01 applications Advice for New Investigators: Apply for R01 Apply Now
31
3. Funding the Best Research Earlier Shorten the Review Cycle More Flexible Deadlines Abolish A2 applications
32
1.Enhanced Review Criteria 2.Template-Based Critiques 3.Scoring Scale (9 point scale) 1.Criterion Scoring 2.Score 4.Clustering and Order of Discussion 4. Improve Quality and Transparency of the Peer Review Process A.Enhanced Review Criteria B.Template based Criteria C.Scoring D.Order of Review E.Training
33
A. Enhanced Review Criteria Overall Impact: ) Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) New Core Criteria Order: Significance Investigator(s) Innovation Approach Environment
34
B. Template-Based Critiques The objective is to write evaluative statements and to avoid summarizing the application Comments should be in the form of bullet points or if necessary short narratives The entire template is uploaded to IAR to become part of the summary statement. Significance 1. Significance Please limit text to ¼ page Strengths Weaknesses
35
C. The Essence of Scoring Before the Meeting Every assigned reviewer will post criteria scores AND overall impact scores. The impact scores will be used to determine the order of discussion. During the Meeting For the discussed applications: The overall impact score is stated by the assigned reviewers. Criteria scores are not mentioned during the discussion. After the Meeting The applicant with a discussed application will see 3 or more criteria scores AND the overall impact score (the one that will be percentiled) The applicants with a non discussed application will see ONLY the criteria scores (3 or more for each criterion)
36
C. Scoring ImpactScoreDescriptor High Impact 1Exceptional 2Outstanding 3Excellent Moderate Impact 4Very Good 5Good 6Satisfactory Low Impact 7Fair 8Marginal 9Poor
37
C. Scoring 720 Applications, 1-9 Scale Number of Applications
38
C. Scoring
39
D. Order of Review Why? Concern of variation of scores during different times of the meeting. The original plan was to recalibrate scores at the end of the meeting n: Solution: Recalibrate dynamically by discussing in order of average preliminary scores from assigned reviewers. Requirement: Reviewers must participate in entire meeting Telephone
40
E. Enhancing Peer Review Training CSR and NIH Review Staff 6 face to face training sessions, January 2009 6 face to face training sessions, April 2009 Continuous updating Chairs For Chairs appointed in 2008, 8 sessions in January 2009 For Chairs appointed in 2009, 7 sessions in April-May 2009 Reviewers Training material (Power Point, interactive training, frequently asked questions, mock study section video, etc,in April-May 2009 Senior CSR staff at the first meeting in May-July 2009
41
Review of ARRA Applications
42
The Stimulus
43
Applications Reviewed by CSR (June-July 2009) Non ARRA Applications (Normal)16,312 Challenges Applications20,894 Competitive Revisions 2,077 GO Grants 2,697 (CSR 500?) P 30 561 TOTAL as of 2/6/0940,000 plus
44
Applications Reviewed and Reviewers Used by CSR in June 2008 and 2009
45
ARRA Applications Under Review RC1. Challenge Grants Deadline April 27, a few still trickling in Received so far 20,894 Verified and Assigned to IRGsall Assigned to 3 Reviewers95% 2 Stages Editorial Board Review oEach application assigned to 3 reviewers (19,000), reviews due by June 5 th, extended to June 12th o30 Special Study Sections in early July oAdditional SS for Late Arrivals and Stem Cells
46
ARRA Applications Under Review Competitive Revisions Deadline April 23, still arriving Large Majority Assigned to the Standard 250 Study Sections Study Sections will review as a SEP in May-June Stem Cells Challenges
47
R01 and R21 Reviewed by CSR 2007 2008 2009
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.