1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

2. CSR Study Section Reorganization Implementation Update Don Schneider January 2005

3. Study Section Reorganization Topics Background – why reorganize Milestones Numbers of applications Shifting basic science applications (to disease/organ IRGs) Assessing peer review in CSR Creation of new study sections

4. Scientific Peer Review at NIH and DRG/CSR 1945-1998: Study sections formed in response to increases in workloads, case-by-case 1998: ADAMHA merger and BDCN, IFCN, & MDCN IRGs formed with community participation 1998/9: AIDS IRG formed 1999: ADAMHA merger and BBBP, HOP, & RPHB IRGs formed with community participation 2000: Panel on Scientific Boundaries for Review/PSBR Report and launching of Phase II systematic reorganization of study sections with community participation

5. PSBR Reorganization Process Phase I: Broad blueprint for peer review philosophy and 24 Integrated Review Groups (IRGs) Phase II: Study section boundaries teams, one for each of 17 IRGs to be reorganized Transparency and assessment

6. Panel on Scientific Boundaries for Review PSBR Principles At least one home for review of all science relevant to health-related research (serve all segments of research community) Research topics of each IRG should be cohesive or scientifically related Organization should be flexible enough to adjust to rapid changes (provide homes for emerging fields)

7. PSBR Goals for Review Process Set high standards Advance health-related science Encourage innovation & risk taking Show fairness & clarity Be monitored continuously

8. Reorganize with a Disease/Organ Focus Mission to improve human health is best served by review in context of human condition Basic studies should be applied to disease problems in timely way Within IRGs, exchange of ideas among reviewers may encourage broader applications Review of basic & applied science in same IRG provides overview of the quality of work Distribution of applications using powerful methodologies enhances application of new methods to disease problems

9. Reorganization and Basic Sciences Clinical advances often rest on results of seemingly unrelated basic research Sizable proportion of basic science ensures rigorous foundation for future progress Much of basic research must be reviewed in a fundamental context without regard to specific disease/organ

10. Cross-Cutting Science No distribution system can achieve perfect clustering for all areas Cross cutting areas need to be clustered Examples: Biology of Development & Aging IRG; transplantation in Immunology IRG; all of chemistry in Biological Chemistry & Macromolecular Biophysics IRG Applications permitting: two review homes are good, e.g., prokaryotic molecular genetics in Genes, Genomes & Genetics IRG & in Infectious Diseases & Microbiology IRG

11. Cultural Norms and Changes Reviewers should be active researchers able to judge scientific merit of broad areas Advocacy for a field should not be a function of peer review (avoid entitlements) Exploratory research or methods development or hypothesis-driven research must be judged on its potential to impact biomedical and behavioral research Overemphasis on preliminary data discriminates against bold new ideas Applicants should participate in referral to study section

12. The Phase II Process Steering Committee of NIH review & program staff for each IRG contacted communities and selected design/Study Section Boundaries Team SSB Team of active researchers & staff (about 30 people) for each IRG designed study sections and wrote guidelines (intense three-day face- to-face meetings, 3x30x16 = 1,440 people days) Ninety days for public comment on study sections and guidelines CSR Advisory Committee considered and recommended study sections and guidelines for each IRG New study section implementation was stepwise over two-year period

13. Panel on Scientific Boundaries for Review (PSBR) Milestone Dates and Events Jan 2000: CSRAC accepted PSBR Phase I report Feb 2001: first SSB Team (HEME) met May 2002: CSRAC recommended first IRG guidelines (HEME) Apr 2003: last SSB Team (CB) met Jun 2003: first study sections (HEME) met Jan 2004: CSRAC recommended last IRG guidelines (BCMB & CB) Feb 2005: last study sections (BCMB & CB) to meet

14. New Study Sections and Numbers of Applications per Cycle Ideal study section workload - 60-80 applications HEME/RES/RUS (9 study sections) Mean65 Median59 BDA/BST (10 study sections) Mean73 Median72 (Data from 2005/01)

15. WORKLOADS IN NEW STUDY SECTIONS SSB Teams overdesigned study sections to be inclusive and broad Sixty-eight study sections averaged about 70 applications in their first meetings Twelve study sections had fewer than 50 applications Fifteen study sections had more than 90 applications

16. CSR Workloads –May Councils 2000 - 2005 CouncilCSR Applications 5/200011,138 5/200110,705 5/200211,870 5/200314,052 5/200416,618 5/200518,038

17. Implementation Update A Disease/Organ IRG Window Study Sections IRG: More clinical/basic/unchanged EMNR: IPOD/MCE/CMIR IDM: CRFS/PCMB/VIRB MOSS: MRS/SMEP/ACTS ONC: CONC/CE/CAMP RUS: PBKD/CMBK/UKGD

18. More Clinical Study Sections IPOD – Integrative Physiology of Obesity & Diabetes CRFS – Clinical Research & Field Study of Infectious Diseases MRS – Musculoskeletal Rehabilitation Sciences CONC – Clinical Oncology PBKD – Pathobiology of Kidney Diseases

19. Basic Study Sections MCE – Molecular & Cellular Endocrinology PCMB – Prokaryotic Cell & Molecular Biology SMEP – Skeletal Muscle Biology & Exercise Physiology CE – Cancer Etiology CMBK – Cell & Molecular Biology of the Kidney

20. Less Changed Study Sections CMIR – Cellular, Molecular & Integrative Reproduction VIRB – Virology B ACTS – Arthritis, Connective Tissue & Skin CAMP – Cancer Molecular Pathology UKGD – Urologic & Kidney Development & Genitourinary Diseases

21. Workloads in Disease/Organ Study Sections (Data from 2005/01) More clinical study sections: 5 @ 76 applications More basic study sections: 5 @ 74 applications Less changed study sections: 5 @ 92 applications

22. Shift of Basic Science Applications to Disease/Organ Study Sections Competitive Renewal Applications Reviewed by Disease/Organ IRGs 2004/10:4/128 = 3% shift 2005/01:15/213 = 7% 2005/05:17/206 = 8% (Preliminary indications at early stages of process)

23. Scoring Fate of Shifted Applications SituationMedian %tilePrevious Dis/org from basic25.5 9.0 Dis/org from diff IRG35.4 11.4 Dis/org from parent IRG40.4 13.5 Basic from basic22.5 7.5 (Data limited to 2005/01)

24. Assessment of IRGs & Study Sections Previous experiences - Working Groups 2000-2002 Every IRG and chartered study section was site visited - Surveys 2001-2003 BDCN, IFCN, & MDCN applicants, reviewers, and staff were surveyed Future issues/variables - Scheduling 5-year cycle Neuroscience IRGs due soon HEME due Feb 2008 - Participation by Scientific communities NIH program and review staff - Reporting to PRAC - Michael Martin is point for planning

25. Timely Adjustments to Changes Changes/Problems Workload increases can exceed the metric of 80 applications a round per SRA, e.g., ONCs 12 regular study sections @93 applications each, with Cancer Etiology at 134; EMNRs Integrative Physiology of Obesity & Diabetes has 123 applications (2005/05) Implementation reveals clustering issues, e.g., DNA repair and G protein applications Science changes, e.g., growth of epidemiology applications in Health of the Population IRG (from 2 to 5 study sections since 2001) Fixes/Solutions Temporary responses may involve forming ad hoc or Special Emphasis Panels If situations are not temporary, within 2-3 rounds, CSR will involve working groups of active researchers and NIH program and review staff in forming regular study section(s) and in developing guidelines Health of the Population IRG formed an Epidemiology Working Group that met via an interactive web site and then face-to-face November 2004 to craft an overall design for 5 epidemiology study sections, with written guidelines

26. Summary of Status of Reorganization Systematic reorganization of study sections was driven by multiple needs (increased breadth, flexibility, openness, etc) Implementation of last study sections is on schedule for February 2005 (98 NEW STUDY SECTIONS, WE HAVE ALMOST DONE IT!) Workloads are still largely reasonable as over design of study sections and increase of applications balance Early results show a shift of applications from basic to disease/organ IRGs (initial scoring results are encouraging) Regular assessment of study sections by reviewers and NIH program and review staff is scheduled A process for timely adjustments to changes by creation of new study sections is in place

27. ACKNOWLEDGEMENTS IMPAC II data and analysis: Teresa Lindquist Charles Dumais Karl Malik