1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

2. Challenges and Opportunities in Peer Review A Vision for Ensuring Its Strategic National Value toni scarpa scarpat@csr.nih.gov 301-435-1109 Peer Review Advisory Committee Rockville, MD June 8, 2009 National Institutes of Health U.S. Department of Health and Human Services

3.  Peer Review at CSR  CSR News  Enhancing Peer review  Review of ARRA Applications Enhancing Peer Review

4.  Peer Review at CSR Enhancing Peer Review

5. $13.7 $15.6 $17.8 $20.5 $23.3 $27.1 $28.0 $28.6 $29.1 $29.5 $0 $5 $10 $15 $20 $25 $30 1998 199920002001 2002 2003 2004 20052006 2007 2008 2009 The NIH Budget Doubling $ In Billions

6. Number of Applications Historical Growth

7. CSR Peer Review: 2008 77,000 applications received 16,000 reviewers 1,400 review meetings 240 Scientific Review Officers

8. CSR Peer Review: 2009 77,000 applications received 16,000 reviewers 1,400 review meetings 240 Scientific Review Officers 115,000 38,000 1,600

9.  Peer Review at CSR  CSR News Enhancing Peer Review

10. 1.Reorganizing CSR and Recruiting Staff 2.Improving Study Section Alignment 3.Assigning Applications more Accurately 4. Shortening the Review Cycle 5. Advancing Additional Review Platforms 6.Special Emphasis Panels 7. Recruiting the Best Reviewers CSR News

11. 1. Reorganizing CSR Translational and Clinical Sci Cardiovascular and Respiratory Sciences Surgical Sciences, Biomedical Imaging and Bioengineering Musculoskeletal, Oral And Skin Sciences Oncology 2 – Translational Clinical Vascular and Hematology Physiological Pathological Sci Endocrinology, Metabolism, Nutrition & Reproductive Sciences Immunology Infectious Diseases & Microbiology Digestive, Kidney & Urological Systems Neuroscience, Development and Aging Brain Disorders & Clinical Neuroscience Molecular, Cellular & Developmental Neuroscience Integrative, Functional & Cognitive Neuroscience Emerging Technologies & Training in Neuroscience Biology of Development & Aging Biobehavioral & Behavioral Processes Risk, Prevention& Health Behavior Population Sciences and Epidemiology Healthcare Delivery & Methodologies AIDS & Related Research AIDS, Behavioral, Population Basic- Integrative Biological Sci Biological Chemistry & Macromolecular Biophysics Bioengineering Sciences & Technologies Genes, Genomes &Genetics Oncology 1 – Basic Translational Cell Biology Interdisciplinary Molecular & Training

12. Three New CSR Division Directors Sy Garte Joy Gibson René Etcheberrigaray Division of Translational & Clinical Sciences Division of Physiological & Pathological Sciences Division of Neuroscience, Development and Aging

13. 1. Reorganizing CSR Translational and Clinical Sci Cardiovascular and Respiratory Sciences Surgical Sciences, Biomedical Imaging and Bioengineering Musculoskeletal, Oral And Skin Sciences Oncology 2 – Translational Clinical Vascular and Hematology Physiological Pathological Sci Endocrinology, Metabolism, Nutrition & Reproductive Sciences Immunology Infectious Diseases & Microbiology Digestive, Kidney & Urological Systems Neuroscience, Development and Aging Brain Disorders & Clinical Neuroscience Molecular, Cellular & Developmental Neuroscience Integrative, Functional & Cognitive Neuroscience Emerging Technologies & Training in Neuroscience Biology of Development & Aging Biobehavioral & Behavioral Processes Risk, Prevention& Health Behavior Population Sciences and Epidemiology Healthcare Delivery & Methodologies AIDS & Related Research AIDS, Behavioral, Population Basic- Integrative Biological Sci Biological Chemistry & Macromolecular Biophysics Bioengineering Sciences & Technologies Genes, Genomes &Genetics Oncology 1 – Basic Translational Cell Biology Interdisciplinary Molecular & Training

14. 2. Improving Study Section Alignment Don Schneider

15. 3. Assigning Applications Accurately

16. 4. Shortening the Review Cycle Why? First Response was 5.2 months, far too long The Goal To review and post score and critique application within 3 months of submission. To enable resubmission, when doable and desirable, 4 months earlier than in the past. The Result Every New Investigator and Most Established Investigators are eligible 13% of those eligible apply

17. 5. Special Emphasis Panels

18. 15% of Applications Percentiling Rosters

19. 6. Advancing Additional Review Platforms Additional Review Platforms Help Recruiting Reviewers Electronic Review Modes Reduce Travel Electronic Reviews Telephone Enhanced Discussions Video Enhanced Discussions Asynchronous Electronic Discussions (AED)

20. 6. Advancing Additional Review Platforms Reviewer Satisfaction with AED Review

21. 6. Advancing Additional Review Platforms Cost Comparison of Review Platforms PhoneAEDVEDFace to Face Cost/application25107237867 Cost/reviewer311002921767 Not including honoraria for reviewers

22. 7. Recruiting the Best Reviewers Some Successful Strategies Moving a meeting a year to the West Coast Additional review platforms Develop a national registry of volunteer reviewers Searchable database with 5,000 reviewers Provide tangible rewards for reviewers No submission deadlines for chartered members of study sections (effective February 2008). Provide flexible time for reviewers Choice of 3 times/year for 4 years or 2 times/year for 6 years

23. 7. Recruiting the Best Reviewers Expansion of No Submission Deadlines Present (since February 2008) Chartered Study Section Members oCSR 3127 oOther ICs10124,139 Planned for 2009 Frequent Reviewers* 1323 BSC regular members 260 NAC members 3931, 976 * 6 meetings/last 18 months

24. Enhancing Peer Review

25. The Process Diagnostic Design Implementatio n Plan Begin Phased Implementatio n of Selected Actions June 2007 – Feb. 2008 March 2008 – June 2008September 2008 2

26. Summary of Recommendations Priority 1. Engage the Best Reviewers Priority 2. Improve Quality and Transparency of Review Priority 3. Ensure Balance Across Scientific Fields and Career Stages Priority 4. Continuous Review of Peer Review

27. Enhancing Peer Review 1. Review Highly Transformative Research 2. Fund Early Stage Career Investigators 3. Fund the Best Research Earlier and Reduce the Burden 4. Improve Quality and Transparency of Peer Review 5. Recruit and Retain the Best Reviewers 6. Train Chairs and Reviewers

28. 1. Review Highly Transformative Research OD Transformative RO1 (T-RO1) Once a year, 5 years, $20-$40 million total budget Deadline January 29, 2009 8-page application 740 applications Editorial Board Review oHeavy triage based on innovation and potential science transformation by a small study section of distinguished, broad-science reviewers (the editors) oSpecific science reviewed by appropriate reviewers (the editorial board) oFinal ranking by the editors (Face to face, June 17,2009)

29. 0% 1% 2% 3% 4% 5% 6% 7% PIs in 1980 Percent of PIs 2. Early Stage Career Investigators Projection of Age Distribution of NIH RPG Investigators: 2020 2530354045505560657075808590 Age

30. 2. Early Stage Career Investigators (ESI) : Definition of New Investigator: Not previously competed successfully as PD/PI for a significant NIH independent research award. Definition of Early Stage Investigator: Within 10 years of completing terminal research degree or within 10 years of completing medical residency (or the equivalent). The NIH corporate policy: to fund R01s of New Investigators and ESIs at different paylines. 3 Paylines for R01s Applies only to R01 applications Advice for New Investigators: Apply for R01 Apply Now

31. 3. Funding the Best Research Earlier Shorten the Review Cycle More Flexible Deadlines Abolish A2 applications

32. 1.Enhanced Review Criteria 2.Template-Based Critiques 3.Scoring Scale (9 point scale) 1.Criterion Scoring 2.Score 4.Clustering and Order of Discussion 4. Improve Quality and Transparency of the Peer Review Process A.Enhanced Review Criteria B.Template based Criteria C.Scoring D.Order of Review E.Training

33. A. Enhanced Review Criteria Overall Impact: ) Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) New Core Criteria Order: Significance Investigator(s) Innovation Approach Environment

34. B. Template-Based Critiques The objective is to write evaluative statements and to avoid summarizing the application Comments should be in the form of bullet points or if necessary short narratives The entire template is uploaded to IAR to become part of the summary statement. Significance 1. Significance Please limit text to ¼ page Strengths  Weaknesses 

35. C. The Essence of Scoring Before the Meeting Every assigned reviewer will post criteria scores AND overall impact scores. The impact scores will be used to determine the order of discussion. During the Meeting For the discussed applications: The overall impact score is stated by the assigned reviewers. Criteria scores are not mentioned during the discussion. After the Meeting The applicant with a discussed application will see 3 or more criteria scores AND the overall impact score (the one that will be percentiled) The applicants with a non discussed application will see ONLY the criteria scores (3 or more for each criterion)

36. C. Scoring ImpactScoreDescriptor High Impact 1Exceptional 2Outstanding 3Excellent Moderate Impact 4Very Good 5Good 6Satisfactory Low Impact 7Fair 8Marginal 9Poor

37. C. Scoring 720 Applications, 1-9 Scale Number of Applications

38. C. Scoring

39. D. Order of Review Why? Concern of variation of scores during different times of the meeting. The original plan was to recalibrate scores at the end of the meeting n: Solution: Recalibrate dynamically by discussing in order of average preliminary scores from assigned reviewers. Requirement: Reviewers must participate in entire meeting Telephone

40. E. Enhancing Peer Review Training CSR and NIH Review Staff 6 face to face training sessions, January 2009 6 face to face training sessions, April 2009 Continuous updating Chairs For Chairs appointed in 2008, 8 sessions in January 2009 For Chairs appointed in 2009, 7 sessions in April-May 2009 Reviewers Training material (Power Point, interactive training, frequently asked questions, mock study section video, etc,in April-May 2009 Senior CSR staff at the first meeting in May-July 2009

41. Review of ARRA Applications

42. The Stimulus

43. Applications Reviewed by CSR (June-July 2009) Non ARRA Applications (Normal)16,312 Challenges Applications20,894 Competitive Revisions 2,077 GO Grants 2,697 (CSR 500?) P 30 561 TOTAL as of 2/6/0940,000 plus

44. Applications Reviewed and Reviewers Used by CSR in June 2008 and 2009

45. ARRA Applications Under Review RC1. Challenge Grants Deadline April 27, a few still trickling in Received so far 20,894 Verified and Assigned to IRGsall Assigned to 3 Reviewers95% 2 Stages Editorial Board Review oEach application assigned to 3 reviewers (19,000), reviews due by June 5 th, extended to June 12th o30 Special Study Sections in early July oAdditional SS for Late Arrivals and Stem Cells

46. ARRA Applications Under Review Competitive Revisions Deadline April 23, still arriving Large Majority Assigned to the Standard 250 Study Sections Study Sections will review as a SEP in May-June Stem Cells Challenges

47. R01 and R21 Reviewed by CSR 2007 2008 2009