Amy Rubinstein, Ph.D. Scientific Review Officer Direct Ranking of Applications: Pilot Study

Applications are assigned to 3 reviewers who provide preliminary impact scores (1-9) and critiques. After panel discussion of each of the top 50% of applications, all panel members vote on a final overall impact score. Each application’s score is derived from the average of all panel members’ votes and multiplied by 10 (resulting in final scores of 10-90). R01 applications are assigned a percentile based on the scores of applications reviewed in the relevant study section in that round and the previous 2 rounds. Current System for Evaluating and Ranking Applications Reviewed in CSR Study Sections

The number of applications reviewed by NIH is at or near historic highs and award rates are at historic lows. It can be difficult to differentiate between the top 1-20% of applications reviewed in study sections. Concerns about the potential for an application to be funded results in compression of scores in the 1-3 range (final scores between 10 and 30). The current system of percentiles is used to rank applications reviewed in different study sections. However, score compression results in many applications with the same percentile, making funding decisions more difficult. Why Consider Direct Ranking?

Distribution of scores for discussed applications

Council RoundScore 20Score 25Score / / / / / / / / / / / / / / / CSR All Percentiles at scores of 20, 25, 30

Andrea Hollingshead, University of Southern California (rank order methods and procedures) Reid Hastie, University of Chicago (behavioral and psychological aspects of ranking and rating systems) David Budescu, Fordham University (statistical considerations when designing scoring and ranking systems) Donald Saari, University of California Irvine (effects of different voting methods on order of preference and how outcomes can be manipulated) Symposium on Ranking: August 2013

Reviewers would not be forced to give applications higher (worse) overall impact scores than they think the applications deserve. Reviewers would be required to distinguish between applications of similar quality and separate the very best from the rest. Reviewers will have an opportunity to re-rank applications after hearing the discussion of all applications, something that is less practical with the current system. Potential Advantages of a Rank Order Method

New Investigators are reviewed in a separate cluster but must be integrated into the final rank order of applications that are reviewed. Applications cannot be ranked with respect to applications in the previous two rounds as is done with the percentile system. Reviewers in study sections that cover highly diverse scientific areas may find direct ranking more difficult. Direct ranking may not allow reviewers to indicate applications of essential equivalent quality. Private ranking may lack the transparency of the current system where reviewers who vote out of the range set by assigned reviewers must provide justification during or after the discussion. Possible Disadvantages of Direct Ranking

Will be carried out in parallel with current review system. Applications will be scored as usual; reviewers will be asked to privately rank the top 10 R01 applications discussed on a separate score sheet. Rank data will be analyzed internally but average rank information will not be shared with Program staff or be used to influence funding decisions. Pilot Study for Direct Ranking of Applications

Average rank, minimum rank, maximum rank for each application. Variance, standard deviation Percentage of eligible reviewers ranking a given application in the top 10. Data Analysis

How does the rank order of directly ranked applications compare to the order obtained by percentiles? Does the rank order method produce fewer ties than the percentile method and therefore better spreading of scores? How much consensus among reviewers is evident in the rank system? (A calculation of variance and the percentage of eligible reviewers ranking an application in the top 10 will be noted.) How do reviewers perceive the new system? Do they find it difficult to remember earlier discussions? Do they find it difficult to judge the relative rank of applications that are of similar quality? (Reviewers will be asked to provide optional, open-ended feedback after the pilot). How might NIH use data from a rank ordering system? (Results will be discussed within CSR and with Program staff using aggregate data. No rank orders for specific applications will be shared with Program Directors). Pilot Evaluations: Questions to Answer

Dr. Ghenima Dirami, SRO, Lung Injury and Repair study section Dr. Tomas Drgon, SRO, Biostatistics Methods and Research study section Dr. Gary Hunnicutt, SRO, Cellular, Molecular and Integrative Reproduction study section Dr. Raya Mandler, SRO, Molecular and Integrative Signal Transduction study section Dr. Atul Sahai, SRO, Pathobiology of Kidney Disease study section Dr. Wei-qin Zhao, SRO, Neurobiology of Learning Memory study section Dr. Adrian Vancea, Program Analyst, Office of Planning, Analysis and Evaluation Direct Ranking Pilot Working Group Members