Evaluation Design Matching Logic Model Selecting Indicators Ben Silliman Youth Development Specialist NC 4-H

Design Concerns Questions to match program goals, levels Indicators of program outcome, quality Questions on program outcome, quality Types of evaluation designs Adjustments to fit audience, conditions Timeline for evaluation activities

Bridging to Program Success How does the evaluation design address each level of the logic model?

Matching Questions to Logic Model Formative Questions Input –Are resources available? –Are constraints addressed? Output –Did activities occur as planned? –Was the target audience served?

Matching Questions to Logic Model Summative Questions Outcome –Were there KASA changes? –Were there practice changes? Impact –Did the cumulative activity lead to change in social, economic, educational, or cultural change?

Matching Questions to Logic Model Bridging to Effects Outcome –Were there KASA changes? Skill in crossing the bridge Confidence in crossing –Were there practice changes? Sustained or advancing skill Assistance to others learning to cross

Matching Questions to Logic Model It’s not about building the bridge Impact –Did the cumulative activity lead to change in social, economic, educational, or cultural change? Confidence under challenge, crisis Grasp of how to build skills Use of skills to improve self and community

Keep your eye on the design Arkansas River through wood slats on the Royal Gorge Bridge, Photo by Eric Gorecki

Indicate exactly where you mean to go Program: Drive from Mecklenburg 4-H to State 4-H Office Outcome Indicator: Arrive at parking lot, 512 Brickhaven

Indicate exactly where you mean to go Program: Travel from Mecklenburg 4-H to State 4-H Office: Process Indicators: Mileage & Turning Points 1: Start out going NORTHWEST on ARMORY DR toward CHARLOTTETOWNE AVE/E INDEPENDENCE BLVD. 0.0 mi… 7: Merge onto I-277 N/NC-16 N via ramp on the LEFT. 1.3 mi 8: Merge onto I-77 N/US-21 N via EXIT 5A toward I- 85/STATESVILLE. 1.8 mi 9: Merge at I-85 N via EXIT 13A toward GREENSBORO mi 10: Take I-40 E toward RALIEGH mi… 17: Turn LEFT onto BRICKHAVEN DR. 0.0 mi 18: End at 512 Brickhaven Dr Raleigh, NC Estimated Time: 2 hours 44 minutes Estimated Distance: miles

Examples of 4-H Indicators Health: Increase in… –Nutrition Knowledge –Exercise –-Number at Healthy Weight Leadership and Citizenship: Increase in: –Civics Knowledge/Skill –Role/Responsibility for Org. or Service

Examples of 4-H Indicators School-to-Work: Increase in… –Science Knowledge or Skill –Career Readiness, Interviewing Skill Life Skills: Improvement in… –Communication Skills –Goal Setting –Research Skills

Examples of 4-H Indicators Program Quality: High levels of… –Safety –Support –Skill Building –Service Norms –-Etc.

Indicators-to-Targets Health: Targeted levels –All participants will score above 80% on Nutrition Knowledge (specify concepts) –All participants will complete __ min. of (specified) Exercise __ times per week –80% of sustained participants will show progress toward/maintenance at Healthy Weight (specify standard)

Targets-to-Design & Questions Health: Question formats –All participants will score above 80% on Nutrition Knowledge (specify concepts) Test participants before and after training Ask participants to point to foods with specific nutritional qualities (e.g, Vitamin C) or to describe qualities of specific foods Observe choices at meal or snack or ask for food recall from previous week

Types of Designs Anecdotal Quasi-Experimental –Single group Experimental –Comparison groups May use recall, retrospective, or pre-post, cross-sectional or longitudinal, quantiative and/or qualitative methods

Health Examples: Types of Designs Anecdotal: Some tell their story –Some participants or others say that/how the program changed eating habits Quasi-Experimental (all participants share) –Participants report (or others observe) changes in knowledge or behavior before-and-after or just after training Experimental (equivalent groups compared) –Participants and comparison groups report changes in knowledge or behavior (or others observe) before-and-after training and often as a follow-up on retention or practice

Targets-to-Types of Designs Health: Types of Designs –Anecdotal –Quasi-Experimental Retrospective, Pre-Post –Experimental Comparison groups May be cross-sectional or longitudinal, quantiative and/or qualitative

Call for the Question If stakeholders want to know… Who participated and how long (input) –Keep enrollment, attendance, activity records Whether the program promoted change –Ask retrospective or pre-post Qs

Call for the Question If stakeholders want to know… How different groups benefit –Keep records & compare group outcomes How much change is due to a program –Use pre-post Qs, comparison groups –Gather additional data about qualities of the group(s) and desired change(s)

Types of evaluation designs Questions on program outcome, quality Adjustments to fit audience, conditions Timeline for evaluation activities