Focusing Your Evaluation A Planning Template

Discerning Readiness Evaluate no program before its time Internal Chemistry Objectives Target program selected from priority area Program concept and practice well-developed Logic Model complete enough to explain to others Objectives reviewed as Specific Measurable Achievable Realistic Time-bound

Discerning Readiness Evaluate no program before its time Internal Chemistry Capable, trained staff Dependable, adaptable Implement program with fidelity Able/willing to accommodate evaluation

Discerning Readiness Evaluate no program before its time Internal Chemistry Recruitment of target audience Consistent, sizeable group Engaged at expected levels of intensity, frequency, duration Time to implement and refine program Integrate evaluation into or beyond activities Use waves of feedback to demonstrate improvements and effects

Discerning Readiness Evaluate no program before its time Internal Chemistry Time and money to conduct evaluation Planning-implementation-evaluation priority reflected in annual and daily plan of work and budget Expertise to plan, gather, analyze data Personal and staff skills, values adequate to perform Appropriate consultants engaged in a timely manner

Discerning Readiness Evaluate no program before its time External Chemistry Clear objectives—understood by stakeholders Stakeholders clear about program objectives Program leader clear about stakeholder expectations Constructive relationships with staff Staff clear about evaluation purpose, procedures Program leader clear about staff expectations and abilities

Discerning Readiness Evaluate no program before its time External Chemistry Reasonable timeline for program development Expectations consistent with research, reason Evaluation questions match program maturity Resources and assistance to evaluate 10% added or accommodated For high-level evaluations (control or comparison groups) receive additional staff, consultants, collaborator assistance (e.g., teachers help with parent consent, data collection)

Discerning Readiness Evaluate no program before its time External Chemistry Support for gathering, analyzing, using data Human resource support: staff time, consulting Technology: hardware and software Organizational effectiveness: coordinated efforts resulting in Greater understanding and support of the program Increased capacity of individuals Strengthened relationships among stakeholders Increased organizational capacity to serve and lead

Determining Purpose Sweeten to taste Stakeholder Perspectives Citizens: Does it make a difference? Short-term Outcomes: How well this Program addresses priority state and local issues Mid-term Outcomes: How this Program builds capacities for learning, leading, relating, and serving Long-term Outcomes: How this Program will improve the status and culture of the community relative to health, education, and economic prosperity

Determining Purpose Sweeten to taste Stakeholder Perspectives Funding Agency: Is it cost-effective? Short-term Outcomes: How the money was used appropriately (consistent with goals) and efficiently (consistent with good management principles) Mid-term Outcomes: How this Program builds capacities for sustaining and expanding learning Long-term Outcomes: How this Program will improve the organization’s ability to manage, innovate, build community capacity, and change systems

Determining Purpose Sweeten to taste Stakeholder Perspectives Parents: Is my child safe? Learning? Short-term Outcomes: How does the Program provide a safe, supportive setting led by high-skill, high-character staff; How well does the Program train youth in areas related to the advertised goals? Mid-term Outcomes: How this Program provide a sequence of learning opportunities and keep him/her interested in belonging Long-term Outcomes: How does this Program help my child get into college or a career

Determining Purpose Sweeten to taste Stakeholder Perspectives Staff: Are we running a quality program? Is this worth my effort? Short-term Outcomes: How does the Program equip me with training and support to do best practices Mid-term Outcomes: How does our training (esp. in evaluation) sequence of learning opportunities and keep him/her interested in belonging Long-term Outcomes: What keeps our Program “state-of-the-Art”?

Determining Purpose Sweeten to taste Stakeholder Perspectives Youth: Are we having a good time? Short-term Outcomes: What does the Program help me learn hands-on and have a good time with friends Mid-term Outcomes: What will I be able to do in another year? (Will it still be fun?) Long-term Outcomes: What keeps our Program “state-of-the-Art”?

Determining Purpose Sweeten to taste Evaluation Standards that shape purpose Utility: How will this evaluation be useful Feasibility: Is evaluation realistic? Propriety: Can this be done ethically? Accuracy: Will the (concept and technical skill) of this evaluation produce Source: The Evaluation Center

Deciding on Level/Type Checking the vintage: Taste test Formative Performance testing needs assessment evaluability assessment structured conceptualization implementation evaluation process evaluation Summative Long-term effects outcome eval. impact evaluation cost-effective/cost- benefit analysis secondary analysis meta-analysis

Deciding on Level/Type Checking the vintage: Taste test Formative Evaluation Ongoing feedback about performance needs assessment to determine target audience, needs, best strategies evaluability assessment to determines feasiblliiy, usefulness structured conceptualization to define the program or technology, the target population, and the possible outcomes implementation evaluation monitors the fidelity of the program or technology delivery process evaluation investigates the process of delivering the program or technology, including alternative delivery procedures Source: Research Methods Knowledge Base

Deciding on Level/Type Checking the vintage: Taste test Summative Evaluation Long-term effects or outcomes of a program outcome evaluations explore whether and how much a program produces targeted results impact evaluation examines the broader effects of the program or technology as a whole cost-effectiveness and cost-benefit analysis address questions of efficiency by standardizing outcomes in dollar costs and values secondary analysis reexamines existing data to address new questions or use methods not previously employed meta-analysis integrates the outcome estimates from multiple studies to arrive at an overall or summary judgement on an evaluation question procedures Source: Research Methods Knowledge Base

Distilling Questions Made to Order Program Quality and Best Practice Safety Support Structure Social acceptance Service-orientation Skill-building Self-efficacy Synergy with family and community

Distilling Questions Made to Order Program Improvement by adjusting Developmental appropriateness Learning Style Topical Focus/Relevance Intensity Frequency Duration

Distilling Questions Made to Order Youth Outcomes (borrow from logic model) Short-term: Knowledge-Attitude-Skill-Aspirations Mid-term: Behavior Change-Application Long-term: Social-Economic-Environment-Cultural Change

Developing a Plan Tapping the Casks Eat Right Indicators Healthy diet Reduced junk food Move More Indicators Flexibility Lung capacity Reduced screen time Healthy weight

Developing a Plan Tapping the Casks Eat Right Indicators Healthy diet Reduced junk food Move More Indicators Flexibility Lung capacity Reduced screen time Healthy weight

Developing a Plan Tapping the Casks Eat Right Indicators Healthy diet Reduced junk food Move More Indicators Flexibility Lung capacity Reduced screen time Healthy weight

Developing a Plan Tapping the Casks Data Sources Child Youth leader Parents Others impacted by child’s work (team members or helpees) Generic data (School records, community

Developing a Plan Tapping the Casks Data Collection Options Test-Survey-Interview Portfolio-Demonstration Observation-Case Study

Developing a Plan Tapping the Casks Data Collection Protocol Ethical Educational Effective

Developing a Plan Tapping the Casks Analyzing data Qualitative: Trends, anecdotes, comments Quantitative: Individual growth, group patterns; correlated elements

Directing Implementation Managing the Vineyard Step-by-Step Procedures Budgeting Timeline Monitoring roles Critique

You can’t find out what you want to know until you know what you want to find.