1. Prevention Education Meeting May 29, 2013 Evaluation 101

2. What is Evaluation?

3. Definitions of Evaluation Examine and judge carefully. Determine value or worth. Process that attempts to determine, systematically and objectively, the relevance, effectiveness, and impact of activities in light of their objectives.

4. Purpose of Evaluation Use evidence to prove that a program has delivered intended services. Show that program has made an impact on the health of a certain population. Windsor, Clark, Boyd, & Goodman. Evaluation of Health Promotion, Health Education, & Disease Prevention Programs

5. Purpose of Evaluation To determine achievement of objectives related to improved health status To improve program implementation To provide accountability to funding agencies, community, and other stakeholders McKenzie, Neiger, & Smeltzer. Planning, Implementing, & Evaluating Health Promotion Programs

6. Strategies/Activities Things that a program is doing that are part of implementation or development (actions) ProcessesTools EventsWorkshops MeetingsConducting training Provide accessDeliver services FacilitateWork with media PartneringCollaborating Develop products, resources, or curriculum

7. Outputs The amount of product or service that the program intends to provide  Type of activity, product or service and target audience ParticipantsClients AgenciesDecision makers CustomersPublic Health care providers Type of evaluation  process evaluation  Can also include satisfaction with activity/product/service

8. Short Term Outcomes Immediate effects of activities including changes in AwarenessKnowledge AttitudesBeliefs ValuesOpinions AspirationsMotivation Skills (knowledge) Type of evaluation  impact evaluation

9. Intermediate Outcomes Changes in behavior Risk factor reductionNorms Self managementPolicies Social actionPractice ProceduresDecision-making Demonstration of skills Type of evaluation  impact evaluation

10. Long Term Outcomes Changes in burden of disease, health status, or quality of life  prevalence or incidence  mortality  also social, economic, civic, environmental factors Type of evaluation  outcome evaluation

11. Data Sources for Evaluation

12. Evaluation Terminology Process evaluation  measurement of activities during implementation Impact evaluation  measurement of immediate observable effects of a program including awareness, knowledge, attitudes, skills, and behaviors Outcome evaluation  measurement of the ultimate goal of a program

13. Process Evaluation Data Quantitative  Counts  # meetings held, # training sessions conducted, # of people who attended  Descriptive Statistics  mean satisfaction rating for event, percent of people who responded they liked the event Qualitative  Descriptive information  description of event, date of event, location of event, description of who attended, what information was included during event

14. Process Evaluation Data Monthly Progress Report Example:  # professional training sessions/workshops conducted  # professionals trained  # informational materials/brochures distributed  # primary prevention education sessions conducted for students  # students attended primary prevention education sessions  # informational materials/brochures distributed  # sessions conducted with faith-based organizations  descriptions of topics covered during sessions

15. Impact Evaluation Data Short term outcomes on a logic model  awareness, knowledge, attitude change data Quantitative  data collected from surveys/questionnaires, written tests Qualitative  data collected from surveys/questionnaires, interviews, focus groups

16. Impact Evaluation Data Intermediate term outcomes on a logic model  behavior change data Quantitative  BRFSS data Qualitative  interviews, focus groups, observations

17. Impact Evaluation Data Example:  The results of the pre/post knowledge tests indicted that participants’ knowledge about healthy relationships increased by 500%.  The results indicated that students learned to use negotiation techniques as assessed by post-test.  Responses on the modeling questionnaire indicate that 85% of students would intervene when other students act disrespectfully.  School administrators at 10 schools strengthened their anti-bullying policies.

18. Outcome Evaluation Data Long term outcomes Changes in health status Quantitative  YRBS data  BRFSS data  Vital data  Hospital Discharge data  ER or ED data Qualitative  Interviews, focus groups

19. Outcome Evaluation Data Example:  2007 YRBS  29.9% of high school students had been in a physical fight at least once in the past year  11.8% of high school students had been hit by their boy/girlfriend in the past year  9.5% of high school students have been forced to have sexual intercourse against their will  2008 BRFSS  12.5% of adult females have been victims of forcible rape  16.7% of adult females have either been raped or have experienced an attempted rape  4.7% of adult males have either been raped or have experienced an attempted rape

20. Evaluation Tools and Resources

21. ABCDE Method of Writing Outcomes A – Audience (Who will change?) B – Behavior (What will change?) C – Condition (By When?) D – Degree (How Much?) E – Evidence (How will the change be measured?)

22. Evaluation Tools Logs  Used to collect process evaluation data (the who, what, when, where, why, how) Knowledge tests  Used to collect impact data Surveys/Questionnaires  Used to collect process, impact, or outcome data Interviews  Used to collect impact or outcome data Focus Groups  Used to collect impact or outcome data

23. Evaluation Activity Developing a data collection tool

24. General Evaluation Guides The Program Managers Guide to Evaluation  http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/repo rts/pmguide/pmguide_toc.html http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/repo rts/pmguide/pmguide_toc.html Kellogg Foundation Logic Model Development Guide  http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf Developing a Logic Model: Teaching and Training Guide  http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.pdf http://www.uwex.edu/ces/pdande/evaluation/pdf/lmguidecomplete.pdf Understanding Evaluation: The Way to Better Prevention Programs  http://www.ed.gov/PDFDocs/handbook.pdf http://www.ed.gov/PDFDocs/handbook.pdf Kellogg Foundation Evaluation Handbook  http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf

25. Topical Evaluation Guides Core Performance Measures for Steps to a HealthierUS  http://www.cdc.gov/eval/steps/Steps%20Core%20Performance%20Measures% 2007-2007.pdf http://www.cdc.gov/eval/steps/Steps%20Core%20Performance%20Measures% 2007-2007.pdf Introduction to Program Evaluation for Comprehensive Tobacco Control Programs  http://www.cdc.gov/tobacco/tobacco_control_programs/surveillance_evaluatio n/evaluation_manual/00_pdfs/Evaluation.pdf http://www.cdc.gov/tobacco/tobacco_control_programs/surveillance_evaluatio n/evaluation_manual/00_pdfs/Evaluation.pdf Physical Activity Evaluation Handbook  http://www.cdc.gov/nccdphp/dnpa/physical/health_professionals/intervention s/handbook.pdf http://www.cdc.gov/nccdphp/dnpa/physical/health_professionals/intervention s/handbook.pdf CDC DASH Evaluation Briefs  http://www.cdc.gov/HealthyYouth/evaluation/resources.htm#4 http://www.cdc.gov/HealthyYouth/evaluation/resources.htm#4 Strategic Planning Kit for School Health Programs  http://www.cdc.gov/HealthyYouth/evaluation/pdf/sp_kit/sp_toolkit.pdf http://www.cdc.gov/HealthyYouth/evaluation/pdf/sp_kit/sp_toolkit.pdf SMART Cards  http://www.cdc.gov/HealthyYouth/evaluation/pdf/SMARTcards.pdf http://www.cdc.gov/HealthyYouth/evaluation/pdf/SMARTcards.pdf

26. Violence Prevention Evaluation Guides Sexual and Intimate Partner Violence Prevention Programs Evaluation Guide  Book by CDC, 2007, can be ordered from  http://wwwn.cdc.gov/pubs/ncipc.aspx http://wwwn.cdc.gov/pubs/ncipc.aspx Measuring Violence-Related Attitudes, Behaviors, and Influences Among Youths: A Compendium of Assessment Tools, Second Edition  http://www.cdc.gov/ncipc/pub-res/pdf/YV/YV_Compendium.pdf http://www.cdc.gov/ncipc/pub-res/pdf/YV/YV_Compendium.pdf

27. Evaluation Websites CDC’s Evaluation Framework  http://www.cdc.gov/eval/framework.htm http://www.cdc.gov/eval/framework.htm Community Tool Box  http://ctb.ku.edu/en/ http://ctb.ku.edu/en/ American Evaluation Association  http://www.eval.org/ http://www.eval.org/ Coalitions Work  http://coalitionswork.com/ http://coalitionswork.com/ Writing Good Goals and Objectives  http://apps.nccd.cdc.gov/dashoet/writing_good_goals/menu.html http://apps.nccd.cdc.gov/dashoet/writing_good_goals/menu.html Basic Guide to Program Evaluation  http://www.managementhelp.org/evaluatn/fnl_eval.htm http://www.managementhelp.org/evaluatn/fnl_eval.htm University of Wisconsin Extension – Program Development and Evaluation  http://www.uwex.edu/ces/pdande/index.html http://www.uwex.edu/ces/pdande/index.html Kellogg Foundation  http://www.wkkf.org/default.aspx?tabid=75&CID=281&NID=61&LanguageID=0 http://www.wkkf.org/default.aspx?tabid=75&CID=281&NID=61&LanguageID=0