Why evaluate? Benefits of program evaluation

YD as EVAL: Webinar Agenda Why evaluate? So what is evaluation? How does evaluation fit youth development? How can evaluation benefit a YD program? What resources are available to learn about evaluation?

What happened at the fair

What difference did the volunteer make?

Benefits of Informal Evaluation Building relationships Gathering facts Offering support, recognition Troubleshooting problems Offering guidance

Evaluation by the ‘net

What difference did it make?

Where was evaluation happening?

Where –else-- was evaluation happening?

Benefits of Formal Evaluation Completing Organizational Evaluation Targeted outcomes and tools Efficient reporting process Data for funding proposals Feedback for improving programs

Who needs to know? Stakeholders Individuals and groups with interest in a program Stakeholders Individuals and groups with interest in a program

Rethinking Evaluation Revitalizing Programs Ask the right questions… …in a user- friendly way… …and put the answers to good use…

So what is Evaluation? Evaluation Critical examination of a program’s resources, activities, and outcomes to judge effectiveness, improve quality and results, and inform decisions. (Patton, 1997)

Evaluation Activities Assessment: Investigating needs and assets Monitoring: Tracking processes and progress Evaluation: Documenting quality and outcomes

Merit: Program Quality and Potential Merit: Quality, Best Practices, potential to promote outcomes Overall: Traits that promote positive development (safety, support, skill-building, self-efficacy… Particular: Afterschool, Camps, Science, Horsemanship  Formative Evaluation: Is the program working as planned?  Summative Evaluation: What program components contribute to positive outcomes

Worth: Program Outcome and Impact Worth Value of a program, grounded in outcomes that accrue to targeted units (individuals, groups, organizations) Short-term: Knowledge, Attitudes, Skills, Aspirations Mid-term: Practices, Knowledge Application, Behaviors Long-term: Sustained impact of the program on Social, Economic, and Environmental conditions

Significance: Importance and priority Significance: Importance of change to program partners Overall: Importance to a broad range of stakeholders Particular: Importance to a particular individual or unit

Evaluation Cycle Preparation Planning--Setting goals, matching strategies Focusing--Refining plans, identifying sources Designing--Developing strategies for collecting data Selecting Methods--Identifying tools to measure for goals Prepara -tion Implen- tation Utiliza- tion

Evaluation Cycle Implementation Collecting Data—gathering info from diverse sources Analyzing Data—reviewing and interpreting data Prepara -tion Implen- tation Utiliza- tion

Evaluation Cycle Utilization Communicating and Using Results—Sharing data about quality and impact; applying findings to improvement Prepara -tion Implen- tation Utiliza- tion

Positive Youth Development and Evaluation (Formal and Informal) Life skill and life-blood of program management –Goal Setting/Project Management –Communication –Problem Solving

Positive Youth Development and Evaluation (Formal and Informal) Life skill and life-blood of program management –Goal Setting/Project Management –Communication –Problem Solving Key strategy in positive youth development –Experiential Learning = Action & Reflection

Youth Development as Evaluation based on Hamilton, Hamilton, & Pittman, 2004 Processes: Child Development knowledge base Principles: Program Theory and Philosophy Practices: Strategies for Programming …and Evaluation Products: Results of PYD Programming

Youth Development as Evaluation based on Hamilton, Hamilton, & Pittman, 2004 Processes: Child Development knowledge base Domains Physical Cognitive Social-Emo Domains Physical Cognitive Social-Emo Stages Early Child Middle Child Adolescence Stages Early Child Middle Child Adolescence Dynamics Maturation Interaction Adaptation Dynamics Maturation Interaction Adaptation

Youth Development as Evaluation based on Hamilton, Hamilton, & Pittman, 2004 Principles: Program theory and philosophy Integrated across Ecosystems Theories of Change Theories of Action Developmentally Appropriate Universal and Inclusive Intentional and Iterative Holistic and Communal Ongoing in Engagement Strengths-based, Risk-aware

Youth Development as Evaluation based on Hamilton, Hamilton, & Pittman, 2004 Practices: Strategies for program implementation Experiential Learning Project-based Inquiry-based Problem-based Adventure-based Participatory Service-learning Experiential Learning Project-based Inquiry-based Problem-based Adventure-based Participatory Service-learning Youth-led/ Y-A partnership Youth-led/ Y-A partnership Safety Support Skill-Opportunities Self-efficacy-building Quality Practices Social Norms Social Inclusion Structure Synergy Safety Support Skill-Opportunities Self-efficacy-building Quality Practices Social Norms Social Inclusion Structure Synergy choices levels choices levels team work team work active listening buddy system

YD as EVAL: Authentic Assessment based on Hamilton, Hamilton, & Pittman, 2004 Practices: Strategies for program implementation …and evaluation Safety Support Skill-Opportunities Self-efficacy-building Quality Practices Social Norms Social Inclusion Structure Synergy Safety Support Skill-Opportunities Self-efficacy-building Quality Practices Social Norms Social Inclusion Structure Synergy choices levels choices levels team work team work active listening buddy system Skills card Expert obs/intx Skills card Expert obs/intx Rubric Trained obs/intx Rubric Trained obs/intx Skills card Expert obs/intx Skills card Expert obs/intx Skills card Check-in/Obs Skills card Check-in/Obs Photo evidence Physical evidence Demo/ Confirmation Demo/ Confirmation

Youth Development as Evaluation based on Hamilton, Hamilton, & Pittman, 2004 Products: Outcomes of programs Character Confidence Competence Caring Contribution Connection

Youth Development as Evaluation based on Hamilton, Hamilton, & Pittman, 2004 Processes: Child Development knowledge base Domains Physical Cognitive Social-Emo Domains Physical Cognitive Social-Emo Stages Early Child Middle Child Adolescence Stages Early Child Middle Child Adolescence Dynamics Maturation Interaction Adaptation Dynamics Maturation Interaction Adaptation

Youth Development as Evaluation based on Hamilton, Hamilton, & Pittman, 2004 Principles: Program theory and philosophy Integrated across Ecosystems Theories of Change Theories of Action Developmentally Appropriate Universal and Inclusive Intentional and Iterative Holistic and Communal Ongoing in Engagement Strengths-based, Risk-aware

YD as EVAL: Minimum Effort Outdoor Skills Program Quality Cabin Group Life Skills Focus Integrate Document Focus Integrate Document Frequency Intensity Duration Frequency Intensity Duration

YD as EVAL: Maximum Effort Cabin Group Life Skills Outdoor Skills Registration Program Quality Parent Survey Parent Survey Agent Focus Groups Agent Focus Groups

Benefits of Evaluation Systematic planning, implementation Monitoring risks, opportunities Accountability and reporting Improving and expanding quality Sustaining, growing funding Support and recognition of partners Recruiting, training new partners Sharing with peers Building broader support, insight

Barriers to Evaluation “We’re too busy… “Evaluation can only hurt us… “We just don’t have the goods…. “We don’t know what to evaluate…

Resources for Evaluation E-Basic Basics Factsheets E-Basic Basics Factsheets

A difference that counts How evaluation makes a fairer fair… Safety…Support…Skill-building Typical/Exceptional care for animals Problem solving about projects, careers Individual resilience and group teamwork  Connectedness  Competence  Confidence

So…Why evaluate? Satisfy our own curiosity… Track quality and impact… Use results to improve, show impact, inform partners, and invite new participants Make the most of our method: action and reflection, learning by doing

Youth Development as Evaluation: Application Safety Support Skill-Opps Self-efficacy-building Social Norms Social Inclusion Structure Synergy Safety Support Skill-Opps Self-efficacy-building Social Norms Social Inclusion Structure Synergy Cabin Group Life Skills Outdoor Skills Program Quality Registration Program Quality Parent Survey Parent Survey

“What I do doesn’t make that much difference…” A Fair Appraisal How did the volunteer promote youth development best practices (safety, support, skill-building, etc.)? How would these qualities over time promote connectedness, competence, confidence, character, caring, and contribution?

Behold the Benefits Professional Level Formal Evaluation Functions Reporting achievements to sponsors and decision-makers Sustaining, improving upon, or enhancing program activities

…and evaluation is experiential learning!

Eval benefits at the Professional Level (Formal evaluation) Accountability to program stakeholders Improvement efforts with staff and volunteers