1 Developing EPA’s Peer Review Program Joint JIFSAN/SRA/RAC Symposium Dorothy E. Patton, Ph.D., J.D. September 30, 2003

2 Overview: Parallel Positions EPA in 1992 –External recommendations – from SAB, GAO –Mixed Policies and Practices All Federal Agencies in 2003 –External Recommendations – from OMB –Assumption: Mixed Policies and Practices

3 Agenda Brief History: Pre-1990 Policy and Practice 1992 Science Advisory Board Report Implementation Themes and Tasks –Re-education –Regularity –Rigor Recommendations Addendum

4 EPA’s Peer Review History Peer Review Practices –Statutory Mandates –Academic Model –Evolution and expansion Initiatives for change –1992 Science Advisory Board Report –1993 Reilly Administration Policy –GAO Report –1994 Browner Administration Policy

Starting Point... “Requir[e] credible, independent peer review of all scientific and technical efforts of ORD and the program and regional offices (including model development and use, data collection and evaluation, monitoring plans, research, technical studies, scoping studies, and assessments.)” EPA Science Advisory Board

6... Message General Concerns –Lack of Review –Non-conforming reviews Specific Concerns –Credible and independent reviews –All scientific and technical efforts –ORD, program offices, regional offices

7 Implementation Program First Generation – learning and testing –Generic template –Office-specific approaches –Models and examples Second Generation – Peer Review Handbook –Multi-office teams –First Edition – initial statement of principles, methods –Second Edition – reflects Agency experience

8 Implementation Tasks Rigor – as to criteria and standards Regularity – as to internal process Re-education – to assure agency-wide implementation

9 Rigor: Criteria & Standards Readiness of Product for Peer Review –Consultation vs. peer review Charge to Peer Reviewers –Specific issues and general invitation Form of Peer Review –Matching form and product Identification of Peer Reviewers Measures of Success

10 Regularity: Process Decisions Coverage –products requiring peer review –Offices conducting peer review Form –Letter reviews –Panel reviews Reviewers –Peer Reviewer independence –Peer Reviewer expertise

11 Regularity: Resources Threshold Considerations –Time allocation –Budget allocation –Staff allocation Staffing Considerations – Value of a “Champion” –“Top down” and “bottom up – Investment in training – Investment in models

12 Re-education Goals –Scientific and regulatory credibility –Enhanced agency product Important Distinctions –Peer review as intermediate vs. end product –Scientific reliability vs. policy preferences –Inquiry vs. endorsement

13 Recommendations: Tangible Considerations Early attention to resource issues Consensus on criteria and standards Focus on Using peer review comments Draw on EPA Handbook, as appropriate Offer models of sound peer review Establish reasonable implementation timelines

14 Recommendations: Intangible Considerations Identify a “champion” Educate and engage managers Tailor to program-specific factors Staff participation and team building Recognize “Change-the-culture” issues Aim for scientific quality

15 Summary: Three “Rs” Rigor – See the Handbook Regularity – See the Handbook Re-education – iterative combination of “top-down” and “bottom-up” approaches

16 Additional Information EPA’s Peer Review Handbook – –EPA 100-B , December 2000 Neutral Science Panels (Federal Judicial Center, 2001) NAS “blue book”

17 Addendum: Post-EPA notes Risk Assessment as a multi-disciplinary technical analysis –Reliance on many different studies –Weight-of-evidence conclusions –Policy choices Regulatory Decision –Risk assessment or other technical analysis –Non-technical considerations

18 Addendum (con’t.) OMB guidance –“scientific and technical Products” –“scientifically rigorous review” –“the science that underlies federal regulation” “study” “Studies that have already been subjected to adequate independent review” “regulatory information... any scientific or technical study... relevant to regulatory policy”

19 Addendum (con’t.) Economics and Social Sciences Submissions from industry, public interest groups, other agencies, academics