The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing

NIH information/guidance Advantages/disadvantages as a reviewer Recommendations for further improvement

Relationship of Old vs. New Scores

NIH Guidance The NIH grant application scoring system uses a 9-point rating for the impact/priority score Assigned reviewers also provide ratings for each review criterion (Significance, Investigator(s), Innovation, Approach, Environment) using the same 9-point scale. –These criterion ratings are provided in the summary statement of all applications, both discussed and undiscussed. –Criterion ratings should be considered in determining the overall impact/priority score, but because the relative importance of each criterion to the overall impact/priority score differs for each application, reviewers should not apply a formula of unweighted or weighted criterion scores across applications.

NIH Guidance Reviewers are strongly encouraged to utilize the full range of the rating scale in determining ratings. Optimally, scores will be normally distributed with very few 1’s and 9’s and a majority of scores in the middle of the range (4-6). Discussed applications will receive impact/priority scores from all eligible (not in conflict) reviewers, and these scores will be averaged and multiplied by 10 to determine the final impact priority score (range of 10 to 90). Because the relative importance of each individual criterion to the overall score differs for each application, reviewers should not use a formula of weighted or unweighted averages across applications to determine the overall impact/priority score.

ScoreDescriptor Additional Guidance on Strengths/Weaknesses 1ExceptionalExceptionally strong with essentially no weaknesses 2OutstandingExtremely strong with negligible weaknesses 3ExcellentVery strong with only some minor weaknesses 4Very GoodStrong but with numerous minor weaknesses 5GoodStrong but with at least one moderate weakness 6SatisfactorySome strengths but also some moderate weaknesses 7FairSome strengths but with at least one major weakness 8MarginalA few strengths and a few major weaknesses 9PoorVery few strengths and numerous major weaknesses Minor: An easily addressable weakness that does not substantially lessen the impact the project Moderate: A weakness that lessens the impact of the project Major: A weakness that is severely limits the impact of the project

ScoreDescriptorSignificance Descriptors 1Exceptional Achieving the proposed aims is likely to advance the research field in profound and lasting ways 2Outstanding Achieving the proposed aims is likely to advance the research field in critically important ways 3Excellent Achieving the proposed aims is likely to advance the research field in important ways 4Very Good Achieving the proposed aims is likely to contribute substantially to the current knowledge base of the research field 5Good Achieving the proposed aims is likely to contribute meaningfully to the current knowledge base of the research field 6Satisfactory Achieving the proposed aims is likely to contribute somewhat to the current knowledge base of the research field 7Fair Achieving the proposed aims is likely to contribute only incrementally to the current knowledge base of the research field 8Marginal Achieving the proposed aims is likely to contribute only minimally to the current knowledge base of the research field 9Poor Achieving the proposed aims is unlikely to contribute in any way to the current knowledge base of the research field

ScoreDescriptorInvestigator Descriptors 1Exceptional The investigators are extremely well qualified to achieve the proposed aims. 2Outstanding 3Excellent 4Very Good The investigators are qualified to achieve the proposed aims. 5Good 6Satisfactory 7Fair The investigators do not appear to have adequate qualifications to achieve the proposed aims. 8Marginal 9Poor

NIH-provided Word Template Specified template provided that reviewers were to use “Limit text to ¼ page” for each of the criterion areas –Significance –Investigators –Innovation –Approach –Environment

Reviewer Advantages Broader range of scores and more descriptions made it somewhat easier to rank applications within each criterion Identifying strengths and weaknesses made the review more focused Review was easier to write in some ways—bullet points of strengths and weaknesses

Reviewer Disadvantages Impact/priority score was still difficult for some applications (e.g. very experienced team, well funded in the past, application extended the work somewhat but not in a very exciting way OR application from relatively new investigator that may move field forward but approach was not precise or defined) Worries about the scope and extent of comments being given with the suggested limitations in the review length

Reviewer Recommendations Need to indicate for the applicants which weaknesses are minor, moderate and major so the applicants can revise accordingly—we were not forced to do this in the review so I worry that the applicants are getting unprioritized comments Recommendations for ¼ page of text is difficult and may not result in higher quality reviews, applications and revisions