Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

Evaluation of Peer Review Pilots National Institutes of Health U.S. Department of Health and Human Services Dr. Andrea Kopstein Director, Office of Planning, Analysis & Evaluation, CSR June 8, 2009 Dr. Andrea Kopstein Director, Office of Planning, Analysis & Evaluation, CSR June 8, 2009

Sample of Pilots Evaluated Two Stage ReviewTwo Stage Review PrebuttalPrebuttal 1-9 Scoring1-9 Scoring

Two Stage Revi Two Stage Review Expected Benefits: Provide specific expertise for a wide range of scientific areas Enable better quality discussion by using a smaller number of stage two reviewers Increase consistency for scoring Emphasize overall significance and impact while preserving the dynamic of reviewer discussion Expand the potential reviewer base. Mail reviewer stage Two stages can simplify management of dyads and conflicts.

Types of Applications in Two Stage Review Pilot Bioengineering Research Partnership (BRP).Bioengineering Research Partnership (BRP). Small Business Innovation Research (SBIR).Small Business Innovation Research (SBIR). ~70% response rate among both Stage 1 and Stage 2 Reviewers

Key Evaluation Findings Willingness to serve as a stage one and/or a stage two reviewer in the future:Willingness to serve as a stage one and/or a stage two reviewer in the future: Majority were willing to participate at either level of review. o82.4%Stage One Reviewers willing to be either a Stage One or a Stage Two Reviewer. o63.9% Stage Two Reviewers willing to be either a Stage One or Stage Two Reviewer. What format reviewers would chose for the review of their own applications:What format reviewers would chose for the review of their own applications: o63.9% of Stage Two reviewers put two-stage review as their first choice and about half of Stage One Reviewers ranked two-stage review as their first choice format of choice for their own applications

Key Evaluation Findings Question regarding having more assigned reviewers per application:Question regarding having more assigned reviewers per application: oMajority felt this review format had a positive effect on the availability of scientific and technical expertise (70.6% of Stage One and 72.2% of Stage Two reviewers). Did Stage One Reviewer critiques inform Stage Two review of applications:Did Stage One Reviewer critiques inform Stage Two review of applications: oOver 2/3rds of Stage Two Reviewers found the Stage 1 critiques to be very helpful. 63.9% Stage Two Reviewers thought this review format allowed them to spend less time reviewing and preparing comments.63.9% Stage Two Reviewers thought this review format allowed them to spend less time reviewing and preparing comments.

Key Evaluation Findings 84.7% Stage One and 69.4% Stage Two reviewers responded that two-stage review format would be appropriate for all grant application mechanisms.84.7% Stage One and 69.4% Stage Two reviewers responded that two-stage review format would be appropriate for all grant application mechanisms.

Editorial Board Reviews: Status SBIRSBIR BRPBRP T-R01T-R01 Challenge GrantsChallenge Grants

SBIR Prebuttal Pilot Meeting 1:Meeting 1: o33 of 47 applications prebuttal submitted (70%) and the prebuttal made a difference in only 2 of the 33 (6%). Meeting 2:Meeting 2: o14 of 18 applications prebuttal submitted (78%) and the prebuttal made a difference for none of the 14 (0%). Only 5 of 14 prebuttals addressed a factual error (36%). Meeting 3:Meeting 3: o17 of 19 applications prebuttal submitted (89%) and the prebuttal made a difference for one of the 17 (6%). Zero of 17 prebuttals addressed a factual error ( 0%). Meeting 4:Meeting 4: o57 of 71 applicants (80%) submitted a prebuttal; only 3 addressed factual errors (5%), with errors all being minor rather than substantive. “The remainder essentially submitted a “mini” Introduction, as if a resubmitted application.”

SRO Prebuttal Evaluation 200 words is too short for a prebuttal: oforces the applicant to use shorthand and the prebuttal becomes less coherent Prebuttal helps good applications: onot badly written applications For meetings with large numbers of applications, the prebuttal concept is too much work -“unworkable.” The prebuttal concept has “public relations (PR) “ value. Applicants make additional promises in their prebuttal submissions and although the SRO told reviewers not to let the additional promises affect scoring, some reviewers may have been affected. The new “bulleted” critiques should result in fewer errors.

Reviewer Prebuttal Evaluation Reviewers who wrote open ended responses were strongly in favor of having this option. Prebuttal option “is a major improvement to the review process.” Stage 2 Reviewers expressed need for a longer prebuttal. oSome remarked that the 200 word limit favored applicants with only one concern. oPrebuttal should be up to a single page in length. Reviewers were concerned about the time allocated for applicants who wanted to submit a prebuttal. oThey are in favor of allowing more than the 3 days (including the weekend) allowed in this pilot for submission of the prebuttal.

How did having a prebuttal opportunity alter the quality of peer review?

Survey Summary of 1-9 Scoring System Pilot 33 Reviewers participated in 2 meetings that piloted 1-9 scoring. o22 completed the survey for a 67% response rate.

Key Evaluation Findings Virtually every question directly pertaining to the 1-9 scoring system revealed positive responses.Virtually every question directly pertaining to the 1-9 scoring system revealed positive responses. o1-9 scoring system should be useful to applicants for interpreting the written comments they receive. oThe new 1-9 scoring system was not deemed to be an additional burden when compared to the old 1-5 scoring system. o100% of respondents indicated they were either “Very Satisfied” (68%) or “Somewhat satisfied” (32%) with the 1-9 scoring system.

How well were you able to communicate the differences in impact or merit of the applications you reviewed using the new 1-9 scoring system?

How helpful were the verbal descriptors (Exceptional to Poor) in your determination of the initial and final scores?

What effect do you feel the scoring of the specific criteria had on helping you determine the overall impact (merit) of the applications reviewed?

What is your overall level of satisfaction with a peer review process that includes the 1-9 scoring scale?

WHAT’S NEXT TR01TR01 NIH External and Internal Constituency Surveys:NIH External and Internal Constituency Surveys:ApplicantsReviewersSROsPOEtc.