It is my business to locate trouble… Making Evaluation Work

Evaluation Cycle beginning at the end Preparation Implemen- tation Utilization

Using Results in many ways with many people who need to know Preparation Implemen- tation Utilization Who needs to know?

Using Results in many ways with many people who need to know Preparation Implemen- tation Utilization What do they need to know?

Using Results in many ways with many people who need to know Preparation Implemen- tation Utilization

Using Results to communicate with decision-making stakeholders Preparation Implemen- tation Utilization Demonstration Advocacy Accountability

Using Results to communicate with broader stakeholders Preparation Implemen- tation Utilization Marketing Informing Recruiting

Using Results to make improvements …with the help of participants Preparation Implemen- tation Utilization Program Improvement Participant Engagement

Using Results to celebrate and encourage staff, volunteers, and participants Preparation Implemen- tation Utilization Participant Recognition

Using Results to build capacity for staff and organizations Preparation Implemen- tation Utilization Training and Supervision Organizational Improvement

Using Results to advance practice Preparation Implemen- tation Utilization Publications Proposals

Using Results in multiple ways Preparation Implemen- tation Utilization

Using Results to write a report Preparation Implemen- tation Utilization Accountability

Report Elements components of a formal report Accountability Opportunity Program Response Impact Problem/Need Research Base -Local Needs -National Trends -Prior Research

Report Elements components of a formal report Accountability Opportunity Program Response Impact Problem/Need Program Details Research Base -Local Needs -National Trends -Prior Research Best Practice Strategy -Audience -Setting -Activities -

Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Research Base -Local Needs -National Trends -Prior Research Best Practice Strategy -Audience -Setting -Activities Changes in Key Indicators relative to context and expectations

Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Research Base -Local Needs -National Trends -Prior Research Best Practice Strategy -Audience -Setting -Activities Changes in Key Indicators relative to context and expectations Why? Who What When Where How What’s so? So What?

Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Research Base 90% of jobs require STEM skills 50% of local teachers cited need for STEM 90% of jobs require STEM skills 50% of local teachers cited need for STEM

Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Best Practice Strategy 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… Research Base 90% of jobs require STEM skills 50% of local teachers cited need for STEM 90% of jobs require STEM skills 50% of local teachers cited need for STEM

Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Best Practice Strategy Changes in Key Indicators 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… Research Base 90% of jobs require STEM skills 50% of local teachers cited need for STEM 90% of jobs require STEM skills 50% of local teachers cited need for STEM Survey: 82% of 345 youth… Obs: 12 teams Excellent on Tech Interview… Records: Service, Training.. Survey: 82% of 345 youth… Obs: 12 teams Excellent on Tech Interview… Records: Service, Training..

Report Elements components of a formal report Try to describe your own program Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Best Practice Strategy Changes in Key Indicators 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… Research Base 90% of jobs require STEM skills 50% of local teachers cited need for STEM 90% of jobs require STEM skills 50% of local teachers cited need for STEM Survey: 82% of 345 youth… Obs: 12 teams Excellent on Tech Interview… Records: Service, Training.. Survey: 82% of 345 youth… Obs: 12 teams Excellent on Tech Interview… Records: Service, Training..

Report Elements components of a formal report Accountability Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Obstacles, Adjustments Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ Conclusions Support Materials Program Details

Report Elements components of a formal evaluation Accountability Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Obstacles, Adjustments Implications Advanced training is needed… Organization policy change… Advanced training is needed… Organization policy change… Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ Conclusions Support Materials Program Details

Report Elements components of a formal evaluation Accountability Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Obstacles, Adjustments Implications Illustrations Advanced training is needed… Organization policy change… Advanced training is needed… Organization policy change… Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ As shown in the Table … One participant said… Pictured in the photo are… As shown in the Table … One participant said… Pictured in the photo are… Conclusions Support Materials Program Details

Report Elements components of a formal evaluation Accountability Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Obstacles, Adjustments Implications Illustrations Advanced training is needed… Organization policy change… Advanced training is needed… Organization policy change… Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ As shown in the Table … One participant said… Pictured in the photo are… As shown in the Table … One participant said… Pictured in the photo are… Conclusions Support Materials Program Details Additional Materials Program History… Organizational Chart… Detailed Budget… Program History… Organizational Chart… Detailed Budget…

Report Elements components of an detailed evaluation So what would you add to your report? Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Problems, Adjustments Implications Illustrations Advanced training is needed… Organization policy change… Advanced training is needed… Organization policy change… Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ As shown in the Table … One participant said… Pictured in the photo are… As shown in the Table … One participant said… Pictured in the photo are… Conclusions Support Materials Program Details Additional Materials Program History… Organizational Chart… Detailed Budget… Program History… Organizational Chart… Detailed Budget…

Evaluation and Research paws for thought Research Discovery of Knowledge Evaluation Evidence for Effectiveness

Evaluability is this the right time to ask? Missing the Mark…or the Data

Focusing what difference was intended? Indicators Level Priority Precision

Big P, little P what did you expect? Big picture Purpose-driven Community-driven Evidence-driven Driven to completion

Mapping Evaluation enjoying the journey Program Plan of Work Logic Model Quality and Outcomes Purposes of Evaluation

Mapping Evaluation enjoying the journey Program Plan of Work Logic Model Quality and Outcomes Purposes of Evaluation

The End Bye

Reporting what do you have to say for yourself? Annual Report: Program Impact