1. 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1

2. 2 Changes to Review Beginning with May/June 2009 Meetings   Enhanced Review Criteria for certain mechanisms   Templates for Structured Critiques   Scoring of Individual Review Criteria – –All applications will receive criterion scores from assigned reviewers   New 1 to 9 Scoring Scale 2

3. 3 Goals of the Changes   Clearer understanding of the basis of application ratings   More emphasis on impact and less emphasis on technical details   Succinct, well-focused critiques that evaluate, rather than describe, applications   Routine use of the entire rating scale 3

4. 4 Before the Review Meeting When reading applications the assigned reviewers should:   Address all applicable criteria and other review considerations   Identify major strengths and weaknesses   Assign scores to each of the 5 “core” criteria   Assign an overall impact/priority score 4

5. 5 Preparation of Critiques When writing your critiques:   Use bulleted points to make succinct, focused comments   Short narratives may occasionally be appropriate, but should be rare   Focus on major strengths and weaknesses (ones that impacted your overall rating of the application) 5

6. 6 Features of Critique Templates   Boxes for evaluating: – –Each core review criterion – –Other applicable review criteria and considerations – –Overall impact of the application   A box for “advice to applicants”   Hyperlinks to web pages providing descriptions of review criteria and additional review considerations 6

7. 7 Excerpt from a Critique Template: Criterion  List major strengths and weaknesses that influenced the overall impact/priority score  Limit text to ¼ page per criterion, although more text may occasionally be needed  Do not enter scores on critiques 1. SignificancePlease limit text to ¼ page Strengths Weaknesses

8. 8 Excerpt from a Critique Template: Protected Form Fields and Drop-downs  Protected elements (Drop-down boxes and form fields) are shaded gray  Part of each template is a PROTECTED form  Reviewers should NOT unprotect the forms!

9. 9 Scoring Individual Review Criteria   There are 5 “core” criteria for most types of grant applications   For example, the core criteria for R01s are: – –Significance – –Investigator(s) – –Innovation – –Approach – –Environment   Use the 9-point scale (1 = exceptional, 9 = poor) for the five “core” review criteria.   Do not enter scores in the critique 9

10. 10 Overall Impact/Priority Scores   Consider criterion strengths and weaknesses of each application in determining an overall impact/priority score   Recognize this is a NEW scoring system and focus on the guidelines for its use   This new scoring system is intended to reflect the “real-world” range of the quality of applications typically seen in actual study sections   It is ESSENTIAL that reviewers take advantage of this unique opportunity to use the entire 1 to 9 range 10

11. Scoring Descriptions ImpactScoreDescriptorAdditional Guidance on Strengths/Weaknesses High 1ExceptionalExceptionally strong with essentially no weaknesses 2OutstandingExtremely strong with negligible weaknesses 3ExcellentVery strong with only some minor weaknesses Medium 4Very GoodStrong but with numerous minor weaknesses 5GoodStrong but with at least one moderate weakness 6SatisfactorySome strengths but also some moderate weaknesses Low 7FairSome strengths but with at least one major weakness 8MarginalA few strengths and a few major weaknesses 9PoorVery few strengths and numerous major weaknesses Non-numeric score options: NR = Not Recommended for Further Consideration, DF = Deferred, AB = Abstention, CF = Conflict, NP = Not Present, ND = Not Discussed Minor Weakness: An easily addressable weakness that does not substantially lessen impact Moderate Weakness: A weakness that lessens impact Major Weakness: A weakness that severely limits impact 11

12. 12 Before Attending the Review Meeting   Post critiques to the Internet Assisted Review (IAR) Web module   Enter criterion scores and overall/priority score in IAR   Do not enter scores as part of the critique! – –Ensures better data integrity – –Allows scores to be placed where needed i.e. Summary Statements, Commons Status – –Makes scores available for future analysis 12

13. 13 IAR: New Drop Down for Five Core Criteria   Reviewers will see new drop-down menus in IAR for entering scores for each criterion New drop-down

14. 14 IAR: Assigned reviewers must submit a critique to upload scores   Reviewers must close the critique file before submitting

15. 15 IAR: Entering Scores and Critiques   Assigned reviewers may not submit Criterion or Preliminary Scores without a critique – –If a reviewer tries to save the criterion and/or preliminary score without uploading the critique, an error message will occur   The maximum file size for a critique is 1 MB

16. 16 IAR: New Header Information in Critique   Preliminary IAR Critique now includes criterion scores

17. 17 IAR: Updating Criterion Scores   Criterion scores can be updated in IAR during the submit phase, edit phase and the final scoring phase   If criterion scores are edited, the PDF of the critique file is regenerated each time because the critique has header information with the criterion scores – –If the criterion scores change, the PDF critique changes

18. 18 IAR: New Popup Listing Criterion Scores   New link on List of Applications screen will display criterion scores for each application View All Scores

19. 19 At the Review Meeting: Procedure for Discussed Applications   Assigned reviewers will discuss strengths and weaknesses of each application – –Recommend overall impact/priority score – –Criterion scores will not be discussed by the committee   All eligible members will record an overall impact/priority score (as is presently true) 19

20. 20 IAR: Edit Criterion Scores on Voter Sheet  Criterion scores can easily be edited by using the voter sheet

21. 21 After the Review Meeting: Updating Scores or Critiques   Assigned reviewers whose opinions changed as a result of discussion at the meeting should use IAR: – –To modify their criterion scores – –To post revised critiques   If criterion scores are edited, the PDF of the critique file is regenerated 21

22. 22 Summary Statements   Overall impact/priority scores of discussed applications will be the average of scores voted by all eligible reviewers, multiplied by 10   Final scores will range from 10-90, in whole numbers   Summary statements for ALL applications will include the criterion scores and critiques posted by assigned reviewers 22

23. 23 For additional information: Enhancing Peer Review at NIH Web Site http://enhancing-peer-review.nih.gov Thank you for your review service 23