1. The Measurement and Evaluation of the PPSI Oregon Pilot Program Paint Product Stewardship Initiative Portland, Oregon December 10, 2009 Matt Keene Office of Policy, Evaluation Support Division United States Environmental Protection Agency Jennifer Nash Director of Policy and Programs Product Stewardship Institute, Inc.

2. Purpose of the Presentation  Provide an overview of the process of integrating measurement and evaluation into the Oregon Pilot Program and describe the work of the evaluation committee

3. Presentation Outline 1.Introduction to Program Evaluation 2.Participatory, Utility-Focused Evaluation 3.Integrating Evaluation into the OR Pilot Evaluation Questions Data and Measures 4.Questions, Comments, Discussion

4. Program Evaluation  Definition A systematic study that uses measurement and analysis to answer specific questions about how well a program is working to achieve its outcomes and why.  Orientation/Approaches to Evaluation Accountability External Audience Learning & Program Improvement Internal/External Audiences  Conduct a program evaluation to gain value- added insight: Learn what works well, what does not, and why Learn how a program could be improved Help inform front-end program design Provide performance information for accountability purposes

5. Measurement and Evaluation Program Evaluation: A systematic study with a definable methodology that uses measurement and analysis to answer specific questions about how well a program is working to achieve its outcomes and why. Performance Measurement: A basic and necessary component of program evaluation that consists of the ongoing monitoring and reporting of program progress, using pre-selected performance measures.

6. The Evaluation Committee: Participatory, Utility-Focused  Participatory, Utility-Focused Evaluation  Evaluation Committee Funding and Support Purpose Progress until now  Evaluation Template Evaluation Questions Audiences Approach Data Sources

7. 1. Team 2. Mission 3. Goals & Objectives 4. Logic Model Building Measurement & Evaluation into the OR Pilot Program PPSI Oregon Pilot Program Program QuestionsDocumentation Measures 1. Context 2. Audience 3. Communication 4. Use 1. Data Sources 2. Collection Methods & Strategy 3. Analysis Tools 4. Data Collection 5. Data Management 1. Evaluation Methodology 2. Evaluation Policy

8.  is our program  Describing the Program  Evaluation Team  Mission  Goals and objectives  Logic model The Program 1. Team 2. Mission 3. Goals & Objectives 4. Logic Model Integrating Evaluation into Program Design

9.  What are the critical questions to understanding the success of the OR program?  What contextual factors may influence the answers to each question?  Who are the audiences for each question? What’s the best way to communicate with each audience? How might each audience use the answer to each question? Evaluation Questions 1. Context 2. Audience 3. Communication 4. Use

10.  What can we measure to answer each question?  Where is the information for each measure?  How can we collect the information? What will be our strategy?  What analytical tools will give us the most useful information?  How will we implement the collection strategy?  How will we manage the data? Measures 1. Data Sources 2. Collection Methods & Strategy 3. Analysis Tools 4. Data Collection 5. Data Management Measures

11. Documentation  Evaluation Methodology  Systematic approach structures a methodology  Evaluation Policy  Guides strategy and planning for evaluation and program management  Method and Policy become part of the OR Program…and the National System? 1. Evaluation Methodology 2. Evaluation Policy Measures Documentation

12. 1. Team 2. Mission 3. Goals & Objectives 4. Logic Model Building Measurement & Evaluation into the OR Pilot Program PPSI Oregon Pilot Program Program QuestionsDocumentation Measures 1. Context 2. Audience 3. Communication 4. Use 1. Data Sources 2. Collection Methods & Strategy 3. Analysis Tools 4. Data Collection 5. Data Management 1. Evaluation Methodology 2. Evaluation Policy

13. General Evaluation Questions  Was the Pilot program collaborative & cooperative?  What is the Paint Stewardship Organization and how did the financing mechanism work?  Did consumer awareness and/or behavior change? Why?  Was the Oregon paint management system environmentally beneficial, economical, and convenient?  What are alternative products and markets for post- consumer paint?  How is the evaluation useful to the rollout of a national paint management system?

14. Measures and Data  Potential Measures Stakeholder ratings of collaboration Inventory of outreach materials Change in consumer recycling behavior Volume collected Cost/unit collected Change in demand for paint products  Data Sources Consumers PPSI OR DEQ PSO  Collection Methods Focus Group Ongoing reporting Document review Survey & Interview

15. Discussion  Challenges ahead Timeframe Baselines Reporting Interpreting results  What is most important to understand about how the program is working?  What are your concerns?

16. Thank You!! Environmental Evaluators Network LinkedIn Group: environmental evaluators network www.nfwf.org/een Jennifer NashMatt Keene 617.236.4853202.566.2240 jennifer@productstewardship.us Keene.Matt@epa.gov Director of Policy and Programs Evaluation Support Division Product Stewardship Institute, Inc. Office of Policy U.S. Environmental Protection Agency