1. Focusing an Evaluation Ben Silliman, Youth Development Specialist NC 4-H Youth Development

2. Keys to Focusing an Evaluation Discerning readiness Determining purpose Deciding on level/type Distilling questions Developing plan Directing implementation

3. Discerning Readiness Evaluate no program before its time Internal Chemistry Clear objectives Capable, trained staff Recruitment of target audience Time to implement and refine program Time and money to conduct evaluation Expertise to plan, gather, analyze data

4. Discerning Readiness Evaluate no program before its time External Chemistry Clear objectivesunderstood by stakeholders Constructive relationships, staff and stakeholders Reasonable timeline for program development Resources and assistance to evaluate Support for gathering, analyzing, using data

5. Determining Purpose Sweeten to taste Stakeholder Perspectives Citizens: Does it make a difference? Funding organization: Is it cost-effective? Parents: Is my child safe? learning? Staff: Are we running a quality program? Youth: Are we having a good time?

6. Determining Purpose Sweeten to taste Evaluation Standards that shape purpose Utility, serving the needs of intended users Feasibility, examining if the evaluation is realistic, prudent, diplomatic, practical Propriety, assessing legal, ethical, in the best interests of all stakeholders Accuracy, technically and broadly conveying the worth of the program

7. Deciding on Level/Type Checking the vintage: Taste test Formative Evaluation Ongoing feedback about performance needs assessment to determine target audience, needs, best strategies evaluability assessment to determines feasiblliiy, usefulness structured conceptualization to define the program or technology, the target population, and the possible outcomes implementation evaluation monitors the fidelity of the program or technology delivery process evaluation investigates the process of delivering the program or technology, including alternative delivery procedures Source: Research Methods Knowledge Base

8. Deciding on Level/Type Checking the vintage: Taste test Summative Evaluation Long-term effects or outcomes of a program outcome evaluations explore whether and how much a program produces targeted results impact evaluation examines the broader effects of the program or technology as a whole cost-effectiveness and cost-benefit analysis address questions of efficiency by standardizing outcomes in dollar costs and values secondary analysis reexamines existing data to address new questions or use methods not previously employed meta-analysis integrates the outcome estimates from multiple studies to arrive at an overall or summary judgement on an evaluation question procedures Source: Research Methods Knowledge Base

9. Distilling Questions Made to Order Program Quality and Process Best practices/Youth experience Program improvement

10. Distilling Questions Made to Order Youth Outcomes Short-term: Knowledge-Attitude-Skill-Aspirations Mid-term: Behavior Change-Application Long-term: Social-Economic-Environment-Cultural Change

11. Developing a Plan Tapping the Casks Selecting Indicators (examples) Health: healthy weight; appropriate exercise Science: skill in soldering, selecting seeds, calf care Citizenship: knowledge/skill in parliamentary procedure

12. Developing a Plan Tapping the Casks Selecting Indicators Health diet/calorie journal meal plan for retreat Science aspirations for learning self-efficacy Citizenship: community service plan time priorities

13. Developing a Plan Tapping the Casks Data Sources Child Youth leader Parents Others impacted by childs work (team members or helpees) Generic data (School records, community

14. Developing a Plan Tapping the Casks Data Collection Options Test-Survey-Interview Portfolio-Demonstration Observation

15. Developing a Plan Tapping the Casks Data Collection Protocol Ethical Educational Effective

16. Developing a Plan Tapping the Casks Analyzing data Qualitative: Trends, anecdotes, comments Quantitative: Individual growth, group patterns; correlated elements

17. Directing Implementation Managing the Vineyard Step-by-Step Procedures Budgeting Timeline Monitoring roles Critique

18. You cant find out what you want to know until you know what you want to find.