Evaluation Basics Purpose of Evaluation As adults and youth design and implement their evaluation, there are several important principles that will help them be more effective…

EvaluationBasics Principles of Evaluation Planning for quality and impact (program development) helps youth leaders… Develop programs strong and focused enough to make a difference Develop programs with long-term impact Develop and evaluate programs in stages, rather than trying to do “too much, too soon”

Evaluation Basics Principles of Evaluation “Dosage,” or the right combination of program activity to produce change in knowledge, skill, or behavior, requires leaders to plan programs with appropriate… Intensity: level of difficulty and amount of content; Frequency: sequence of learning experiences, and Duration: length of time …to bring about the change they aim to achieve

Evaluation Basics Principles of Evaluation The importance of intensity, frequency, and duration are illustrated in this example: Pre-teens given 2 hrs/day of guided hands-on practice with a GPS device in a week-long Day Camp learned basic skills for operating the device; Similar peers who shared time on GPS devices did not learn basic skills Note on intensity: Extensive guidance and personal practice is more likely to produce results than more limited guidance and practice (but 4 hrs/day might be too much!)

Evaluation Basics Principles of Evaluation The importance of intensity, frequency, and duration are illustrated in this example: Overweight teens who were trained and tracked to follow a weight-reduction diet, changed eating habits and decreased weight during the six- week project but did not sustain results three months later Notes on duration: Results suggest that the first phase was long enough to produce expected results Whatever happened after the second phase was not enough to sustain results

Evaluation Basics Principles of Evaluation Programs are most likely to make a change in knowledge, skill, or behavior, programs through… Evidence-based curricula, shown to produce specific results with a similar audience, or Research-based practices, shown to promote generally positive effects with a similar audience… …but program fidelity, or consistency with the original model is critical for achieving similar outcomes

Evaluation Basics Principles of Evaluation Key questions in choosing curricula and practices: What is the research evidence for this program? How does that evidence match my program goals, audience, or conditions?

Evaluation Basics Principles of Evaluation Key questions in using curricula and practices: How closely am I following curriculum or practice guidelines (e.g., for intensity, frequency, duration)? If I am not following the recommended guidelines do I expect the advertised results? …program fidelity or consistency with the original model is critical for achieving similar outcomes

Evaluation Basics Principles of Evaluation Programs are most likely to alter knowledge, skill, or behavior when targeted to the learner’s readiness for change, as described in these stages: Pre-contemplation: Never thought about change Contemplation: Aware, but not considering change Preparation: Willing to try change; experimenting Action: Trying to make changes for 3-6 months Maintenance: Sustaining changes beyond 6 months Relapse: Breakdown of change efforts; need to re-try

Evaluation Basics Principles of Evaluation A dropout prevention program targeted to the right level of change for teens interested in automobiles… Youth may be prepared for Automotive program but have not contemplated a Plan-for-College program  Positive hands-on experience with a caring, expert automotive trainer may lead to contemplation of science, technology, and math courses (and staying in school)

Evaluation Basics Principles of Evaluation Middle School youth with strong science skills may be ready for applied programs that maintain or extend knowledge and skills… …but science buffs may consider a (pre)contemplation “Making Science Fun” program too simple and boring  Two-way communication in programs is critical to balancing challenge and support and tapping youths’ readiness to learn, serve, and lead

Evaluation Basics Principles of Evaluation Planning for quality and impact (program development) helps youth leaders… Develop long-term (high-impact) Programs rather than focus on short-term (lower-impact) activities

Evaluation Basics Principles of Evaluation A long-term, multi-faceted Program typically produces deeper and broader results than one or more isolated “programs” or activities A year of planned science, service, and life skills experiences usually results in more learning and applied practice than a series of unplanned events Repeated participation in the same event, such as Presentation Contest or camp promotes more learning than random participation in unrelated events

Evaluation Basics Principles of Evaluation Programs mature over time, and so should evaluations: Newly-initiated programs benefit from Formative Evaluation tracking audience, goals, and quality Established programs can begin Summative Evaluation including pre/post measures of outcomes Mature programs are able to conduct more extensive evaluations of quality and outcomes, comparing results with matched groups

Evaluation Basics Principles of Evaluation Program maturation refers to the level of operation achieved in a youth program. Evaluation goals and methods vary depending on the needs and capacities of program staff, participants, and stakeholders at any given time.

Evaluation Basics Principles of Evaluation Keeping in mind the basic principles for program and evaluation success, leaders of youth programs can begin learning and using evaluation practices…