Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files. archivedOER Public Archive Home PagearchivedOER Public Archive Home Page

History of Priority Scores at the NIH Michael R. Martin, Ph.D. Director, Division of Physiology and Pathology Center for Scientific Review National Institutes of Health June 30, 2005

In the Beginning: August 1946 DRG panels patterned on Office of Scientific Research and Development /Committee on Medical Research review panels DRG panels patterned on Office of Scientific Research and Development /Committee on Medical Research review panels Panel votes to either approve or disapprove Panel votes to either approve or disapprove

And Then: January 1949 A substantial increase in the number of applications and approval ratings leads to a priority rating system A substantial increase in the number of applications and approval ratings leads to a priority rating system Use a 1 – to – 5 scale in whole digit intervals, along with rating criteria to establish order of payment. Use a 1 – to – 5 scale in whole digit intervals, along with rating criteria to establish order of payment. Scores were averaged and multiplied by 100. Scores were averaged and multiplied by 100. Rating criteria standardized Rating criteria standardized

1960s 1962: Institute review panels established 1962: Institute review panels established B/I/Ds tested a variety of procedures for rating including 3 and 10 point scales, separate ratings for scientific merit and rank order, and tenth point intervals in priority scores. B/I/Ds tested a variety of procedures for rating including 3 and 10 point scales, separate ratings for scientific merit and rank order, and tenth point intervals in priority scores. Different DRG study sections began to show different scoring patterns, raising questions about comparability Different DRG study sections began to show different scoring patterns, raising questions about comparability

1971: Executive Committee on Extramural Activities Priority Score Review Committee Recommend retaining priority score Recommend retaining priority score Recommend one standard procedure for adjusting score: suggest normalization or percentiles Recommend one standard procedure for adjusting score: suggest normalization or percentiles 1972: one year pilot of normalization 1972: one year pilot of normalization

1973: Normalization Evaluation Committee Evaluation finds that normalization is broadly accepted, but three ICs object Evaluation finds that normalization is broadly accepted, but three ICs object Normalization discontinued Normalization discontinued

Grants Peer Review Committee 1976 Some ICs continue to use normalization and some raw scores. Some ICs continue to use normalization and some raw scores. Recommended a single notation and score convention Recommended a single notation and score convention

1979: NIH Committee to Study Priority Scores Recommended: Recommended: Displaying only the normalized score Displaying only the normalized score Reviewer should use 0.1 point intervals [primarily because of reviewer insistence and the fact it wouldn't make any difference to the outcome] Reviewer should use 0.1 point intervals [primarily because of reviewer insistence and the fact it wouldn't make any difference to the outcome] Score should continue to be displayed to three digits [in part because two would cause confusion] Score should continue to be displayed to three digits [in part because two would cause confusion] Two ICs object to normalization on the grounds that normalization would obliterate the distinctions between the more innovative fields and those which are less innovative. Two ICs object to normalization on the grounds that normalization would obliterate the distinctions between the more innovative fields and those which are less innovative.

1980 Frederickson mandated the use of raw priority scores but allowed ICs to develop alternatives. Tenth point intervals accepted. Frederickson mandated the use of raw priority scores but allowed ICs to develop alternatives. Tenth point intervals accepted. For FY 1980 Congress mandated a payline of 212, thus adjusting downward the budget of some ICs that used only normalized scores. For FY 1980 Congress mandated a payline of 212, thus adjusting downward the budget of some ICs that used only normalized scores.

Priority Scores in the Early 1980s The pressure between number of applications and appropriation levels increases. The pressure between number of applications and appropriation levels increases. Major priority score aberrations begin: Major priority score aberrations begin: Mean priority score dropped from 250 in 1960s to below 200 in 1985 Mean priority score dropped from 250 in 1960s to below 200 in 1985 “A” and “B” study sections have very different scoring patterns even though they review the same types of applications “A” and “B” study sections have very different scoring patterns even though they review the same types of applications

EPMC Working Group on the Movement of Priority Scores, 1987 Recommendation #1: adopt percentiling system wide Recommendation #1: adopt percentiling system wide Recommendation #2: return to the convention of using 9 steps in the 1 to 5 scale [steps of 0.5] Recommendation #2: return to the convention of using 9 steps in the 1 to 5 scale [steps of 0.5] Recommended that two digits be displayed instead of three, or three digits with the last digit rounded to zero Recommended that two digits be displayed instead of three, or three digits with the last digit rounded to zero

Committee on Rating of Grant Applications(1996) Recommendations: Recommendations: Applications evaluated on three criterion Applications evaluated on three criterion Score on each Criterion Score on each Criterion Reviewers use an eight step scale (0-7) scale Reviewers use an eight step scale (0-7) scale Scores are averaged and reported as two digits Scores are averaged and reported as two digits

Committee on Rating of Grant Applications(1996) PROG and DRGAC discussion: PROG and DRGAC discussion: Propose four criteria instead of three Propose four criteria instead of three Criteria should be specifically addressed in critique Criteria should be specifically addressed in critique Written critique should drive message, not scores on individual criteria Written critique should drive message, not scores on individual criteria

Committee on Rating of Grant Applications(1996) PROG and DRGAC discussion: PROG and DRGAC discussion: Criterion scores not supported Criterion scores not supported Considerable overlap [or lack of distinction?] between the individual criteria Considerable overlap [or lack of distinction?] between the individual criteria Correlation between criterion would be very high, diminishing the value of an individual score Correlation between criterion would be very high, diminishing the value of an individual score Giving individual criteria scores would detract from the message in the critique. Giving individual criteria scores would detract from the message in the critique.

Committee on Rating of Grant Applications(1996) PROG and DRGAC discussion: PROG and DRGAC discussion: Global score should be retained Global score should be retained Taps scientific expertise and judgment of reviewer Taps scientific expertise and judgment of reviewer An algorithm would limit reviewers flexibility An algorithm would limit reviewers flexibility Criterion weights would change project by project Criterion weights would change project by project

Committee on Rating of Grant Applications(1996) PROG and DRGAC discussion: PROG and DRGAC discussion: Cool to reduced rating scale Cool to reduced rating scale Current scheme accepted and understood Current scheme accepted and understood Unscored range reduces the number of bins to less than 20 Unscored range reduces the number of bins to less than 20 No evidence that experts cannot distinguish between more than 11 bins No evidence that experts cannot distinguish between more than 11 bins

Committee on Rating of Grant Applications(1996) Approved: Approved: Five criterion Five criterion Single global score (1-5 scale in tenths) Single global score (1-5 scale in tenths) Criterion-based discussion and critiques Criterion-based discussion and critiques Not Approved: Not Approved: Scoring by criterion Scoring by criterion Eight interval scale and two integer average Eight interval scale and two integer average

Other Studies 1974: RAND Corporation study shows that funded applications with good scores produced more highly cited publications 1974: RAND Corporation study shows that funded applications with good scores produced more highly cited publications 1988: DRG study comparing half point to one tenth point priority score intervals with verbal descriptors suggests that there is little value in the half point scale compared to the tenth point scale. Reviewers react very negatively to the more course scale. 1988: DRG study comparing half point to one tenth point priority score intervals with verbal descriptors suggests that there is little value in the half point scale compared to the tenth point scale. Reviewers react very negatively to the more course scale.

Other Studies 1991: DRG study of re-ranking applications at the end of a meeting shows little impact on outcome 1991: DRG study of re-ranking applications at the end of a meeting shows little impact on outcome 1995: NIGMS begins using rounded priority score. Stops after two years because of Advisory Committee concerns that valuable information is lost. 1995: NIGMS begins using rounded priority score. Stops after two years because of Advisory Committee concerns that valuable information is lost.

Observations Utility and value of priority score method is widely recognized. Utility and value of priority score method is widely recognized. Reviewers report being very comfortable with 41 interval scale. Reviewers report being very comfortable with 41 interval scale. Priority scores are reliable indicators of relative scientific merit and rank order within a study section over time but not between study sections. Priority scores are reliable indicators of relative scientific merit and rank order within a study section over time but not between study sections. Percentile is a valuable tool that has allowed cross study section comparisons. Percentile is a valuable tool that has allowed cross study section comparisons.

History of Priority Scores at the NIH Michael R. Martin, Ph.D. Director, Division of Physiology and Pathology Center for Scientific Review National Institutes of Health June 30, 2005