1. It is my business to locate trouble… Making Evaluation Work

2. Evaluation Cycle beginning at the end Preparation Implemen- tation Utilization

3. Using Results in many ways with many people who need to know Preparation Implemen- tation Utilization Who needs to know?

4. Using Results in many ways with many people who need to know Preparation Implemen- tation Utilization What do they need to know?

5. Using Results in many ways with many people who need to know Preparation Implemen- tation Utilization

6. Using Results to communicate with decision-making stakeholders Preparation Implemen- tation Utilization Demonstration Advocacy Accountability

7. Using Results to communicate with broader stakeholders Preparation Implemen- tation Utilization Marketing Informing Recruiting

8. Using Results to make improvements …with the help of participants Preparation Implemen- tation Utilization Program Improvement Participant Engagement

9. Using Results to celebrate and encourage staff, volunteers, and participants Preparation Implemen- tation Utilization Participant Recognition

10. Using Results to build capacity for staff and organizations Preparation Implemen- tation Utilization Training and Supervision Organizational Improvement

11. Using Results to advance practice Preparation Implemen- tation Utilization Publications Proposals

12. Using Results in multiple ways Preparation Implemen- tation Utilization

13. Using Results to write a report Preparation Implemen- tation Utilization Accountability

14. Report Elements components of a formal report Accountability Opportunity Program Response Impact Problem/Need Research Base -Local Needs -National Trends -Prior Research

15. Report Elements components of a formal report Accountability Opportunity Program Response Impact Problem/Need Program Details Research Base -Local Needs -National Trends -Prior Research Best Practice Strategy -Audience -Setting -Activities -

16. Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Research Base -Local Needs -National Trends -Prior Research Best Practice Strategy -Audience -Setting -Activities Changes in Key Indicators relative to context and expectations

17. Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Research Base -Local Needs -National Trends -Prior Research Best Practice Strategy -Audience -Setting -Activities Changes in Key Indicators relative to context and expectations Why? Who What When Where How What’s so? So What?

18. Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Research Base 90% of jobs require STEM skills 50% of local teachers cited need for STEM 90% of jobs require STEM skills 50% of local teachers cited need for STEM

19. Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Best Practice Strategy 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… Research Base 90% of jobs require STEM skills 50% of local teachers cited need for STEM 90% of jobs require STEM skills 50% of local teachers cited need for STEM

20. Report Elements components of a formal report Accountability Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Best Practice Strategy Changes in Key Indicators 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… Research Base 90% of jobs require STEM skills 50% of local teachers cited need for STEM 90% of jobs require STEM skills 50% of local teachers cited need for STEM Survey: 82% of 345 youth… Obs: 12 teams Excellent on Tech Interview… Records: Service, Training.. Survey: 82% of 345 youth… Obs: 12 teams Excellent on Tech Interview… Records: Service, Training..

21. Report Elements components of a formal report Try to describe your own program Evidence of Merit and Worth Opportunity Program Response Impact Problem/Need Program Details Best Practice Strategy Changes in Key Indicators 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… 500 youth…50 events… 60 volunteers Partnering with schools, businesses, civic clubs… Research Base 90% of jobs require STEM skills 50% of local teachers cited need for STEM 90% of jobs require STEM skills 50% of local teachers cited need for STEM Survey: 82% of 345 youth… Obs: 12 teams Excellent on Tech Interview… Records: Service, Training.. Survey: 82% of 345 youth… Obs: 12 teams Excellent on Tech Interview… Records: Service, Training..

22. Report Elements components of a formal report Accountability Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Obstacles, Adjustments Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ Conclusions Support Materials Program Details

23. Report Elements components of a formal evaluation Accountability Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Obstacles, Adjustments Implications Advanced training is needed… Organization policy change… Advanced training is needed… Organization policy change… Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ Conclusions Support Materials Program Details

24. Report Elements components of a formal evaluation Accountability Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Obstacles, Adjustments Implications Illustrations Advanced training is needed… Organization policy change… Advanced training is needed… Organization policy change… Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ As shown in the Table … One participant said… Pictured in the photo are… As shown in the Table … One participant said… Pictured in the photo are… Conclusions Support Materials Program Details

25. Report Elements components of a formal evaluation Accountability Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Obstacles, Adjustments Implications Illustrations Advanced training is needed… Organization policy change… Advanced training is needed… Organization policy change… Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ As shown in the Table … One participant said… Pictured in the photo are… As shown in the Table … One participant said… Pictured in the photo are… Conclusions Support Materials Program Details Additional Materials Program History… Organizational Chart… Detailed Budget… Program History… Organizational Chart… Detailed Budget…

26. Report Elements components of an detailed evaluation So what would you add to your report? Opportunity Program Response Impact Challenges New Opportunities Enhancements Appendices Problems, Details, Evidence Problems, Adjustments Implications Illustrations Advanced training is needed… Organization policy change… Advanced training is needed… Organization policy change… Difficult Experiences 30% had no transportation… When the budget was cut…\ 30% had no transportation… When the budget was cut…\ As shown in the Table … One participant said… Pictured in the photo are… As shown in the Table … One participant said… Pictured in the photo are… Conclusions Support Materials Program Details Additional Materials Program History… Organizational Chart… Detailed Budget… Program History… Organizational Chart… Detailed Budget…

27. Evaluation and Research paws for thought Research Discovery of Knowledge Evaluation Evidence for Effectiveness

28. Evaluability is this the right time to ask? Missing the Mark…or the Data

29. Focusing what difference was intended? Indicators Level Priority Precision

30. Big P, little P what did you expect? Big picture Purpose-driven Community-driven Evidence-driven Driven to completion

31. Mapping Evaluation enjoying the journey Program Plan of Work Logic Model Quality and Outcomes Purposes of Evaluation

32. Mapping Evaluation enjoying the journey Program Plan of Work Logic Model Quality and Outcomes Purposes of Evaluation

33. The End Bye

34. Reporting what do you have to say for yourself? Annual Report: Program Impact