1. Session 853 Extending Organizational Capacity and Capability to Evaluate Federal Environmental Research Programs Howard Cantor U.S. EPA Office of Research and Development October 29, 2005 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association Session 853 Extending Organizational Capacity and Capability to Evaluate Federal Environmental Research Programs Introduction Howard Cantor U.S. EPA Office of Research and Development October 29, 2005 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association

2. 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association Session Focus  Question: How can we develop an evaluation framework that helps organizational leaders to  Value and use evaluation results;  Build evaluation capability & capacity and  Communicate about performance, outcomes, and evaluation to our clients partners, and stakeholders

3. 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association Why Is this Important?  Growing emphasis on evaluation & accountability for federal research programs: –Canada, European Union, Australia, New Zealand, Japan, Korea, United States

4. 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association Why Is this Important?  Challenges to evaluating federal R & D programs and to measuring performance:  Measuring new knowledge as it develops  Managing the pace at which research progresses  Coping with an absence of accurate methods to objectively evaluate research quality and impact  Reaching agreement about key evaluation definitions  Communicating research contributions to outcomes in language that the public understands

5. 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association Why Is this Important?  Within the United States, EPA program managers and agency leaders must communicate to many stakeholders about accountability:  The Government Performance and Results Act  The President’s Management Agenda  The OMB Program Assessment Rating Tool (PART)  The Research and Development Investment Criteria  The Information Quality Act

6. 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association Specific Evaluation Objectives for Research Programs  Within the U.S., the Research & Development Investment Criteria focus specifically on research program accountability  Within the U.S., the Research & Development Investment Criteria (relevance, quality, performance) focus specifically on research program accountability R&D investments must be planned to be relevant to national priorities, agency missions, strategic goads & customer needs Programs must maximize the quality of the research they invest in R&D programs must demonstrate performance by setting annual and long-term goals and demonstrating progress toward outcomes   OMB and OSTP also encourage federal research managers to characterize the scientific leadership of their research programs

7. 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association Today’s Presentations Focus on: Dale Pahl 15 minutes Lori Kowalski 20 minutes David Schmeltz 30 minutes 20 minutes 1.Integrating program design, management, accountability, & evaluation: conceptual framework 2.Independent expert evaluation with a FACA panel: the Board of Scientific Counselors 3.Research contributions to outcomes: the acid rain cap & trade program Discussion

8. 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association Thanks to our Session Sponsors !  Research, Technology, and Development Evaluation Group  co-chairs: Gretchen Jordan and George Teather  Environmental Evaluation Group  co-chairs: Kara Crohn, Katherine Dawes, Allison Titcomb