Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archived OER Public Archive Home Page

2 PEER REVIEW ADVISORY COMMITTEE Clustering of Grant Applications for Review Don Schneider, Ph.D. [Michael Martin, Ph.D. and Dana Plude, Ph.D.] National Institutes of Health U.S. Department of Health and Human Services June 8, 2009

3 Agenda Peer Review: balancing breadth & depth Evaluate clustering/orphan issue/perception Analyze clustering/orphan data Consider options

4 Clustering of applications raises how-to questions Promote optimal fairness Provide maximum competition Encourage new, emerging science

5 PRAC recommended comprehensive study (Dec 2007) PI Perspective - “I don’t know any of the people on the roster” Databases were analyzed systematically PRAC said - “just ask – staff can tell you” Review & program staffs were polled

One definition of orphan applications Idea – not mainstream in the study section Definition - orphan applications of a given IC are those that are less than 5% of the applications in a study section 6

8

Numbers of IC orphan applications With 28,642 R01 applications reviewed by CSR in FY07: About 2,226 R01s are “IC orphans”, 8% of total Or about 700 applications a cycle 9

Review and Program Staff Members Polled 175 SROs polled directly, 26 SROs identified 27 orphans: 16 ICs [AI, DK, ES & HL 3; remainder one each, including CA & GM] Five orphans per st sx: 60% SROs said five is high; 17% said five is low 500 POs polled indirectly, 17 POs identified 22 orphans No orphan applications were reported simultaneously by both review and program 10

Subsequent review outcomes (from SRO poll) Number unscored: overall 56% UN; however, of T1A0 only 8% scored, and, of T1A2, 100% scored T1 vs T2: majority T1, 88% UN resubmissions: two-thirds have not resubmitted 11

NIH 20%tile or Better Scoring (R01s, CSR reviews) Councils New Applications Established Investigators (Type 1) Renewals From Established Investigators (Type 2) May %34% May %34% May %33% May %34% May %33% May %34% May %32% May %32% New projects score below average CSR Established Investigators Score%

Resubmission Rates for Unsuccessful A0 Projects vary [Dana Plude]

14 Interim conclusions are possible: A small percentage of applications are SRO orphansA small percentage of applications are SRO orphans Every IC has orphan applications; small ICs tend to have higher percentageEvery IC has orphan applications; small ICs tend to have higher percentage New applications tend to score below average, especially T1A0 applicationsNew applications tend to score below average, especially T1A0 applications

Consider review options for orphan applications Business as usual – no change is necessary Create a new study section for orphan area (need 50/more applications) Create IRG/division-wide clusters (in one or more study sections) and maintain breadth of study section(s) 15

Narrowly-defined study sections are problematic Grants Administration Manual (GAM): NIH policy requires that: peer review be carried out in a manner that ensures objectivity, fairness, and maximum competition Scientifically broad-based study sections enable continual advancement and support of new and emerging areas 16

PROPOSAL: Cluster and cross-assign when reviewing orphan applications Group orphan applications to create cluster(s) while maintaining breadth of study section(s) Avoid lone reviewer in orphan area, not sufficient Recruit minimum of 4-5 reviewers in orphan area who also have expertise in other areas reviewed by study section – integration and cross-assignments are vital to preventing camps Have reserve pool of reviewers 17

Clustering/cross-assigning has merits Shows responsiveness Maintains study section breadth necessary for maximum competition and for encouraging new science Lessens “I don’t know any of the people on the roster” comments 18

19 Revisit Agenda Peer Review: balancing breadth & depth Evaluate clustering/orphan issue/perception Low frequency, 2-8% of applications Analyze clustering/orphan data Correlation with T1A0 Consider options Proposal – cluster & cross-assign

20 Acknowledgement PRAC – Clustering Working Group [met 11/30/2007; 6/2/2009] Noni Byrnes, CSR Patricia Greenwel, CSR Ann Hagan, PRAC, NIGMS Leslie Leinwand, PRAC, Colorado Don Schneider, CSR Ross Shonat, CSR Phil Smith, NIDDK

Discussion

22 SUMMARY - Comments from CSR SROs Orphan topics include: sociocultural factors affecting bone density, circadian rhythms, Archaea, sleep disorders, immunology of larynx, vision & driving, blind navigation, transcranial magnetic stimulation, toxic agents and pregnancy, toxic agents and heart, massage therapy, host defense in transplantation, modeling of wound healing, tobacco toxicology, sepsis, alternative medicine, pain, monkeys, pure theory such as modeling of brain oscillations, animal communications, motion sickness, gastrointestinal epidemiology, surfactant biophysics, enzyme nitrosylation, neuro-AIDS and end-organ diseases like enteritis, AIDS and rare opportunistic infections, models to test toxicity, prevention of diabetic foot ulcer, brain vascular development, Hashimoto’s thyroiditis, EPR imaging