Program Planning & Evaluation Begin with the End in Mind Dr. Dallas L. Holmes Specialist Institutional Research Utah State University Extension

Program Planning Process Report Results to Stakeholders Redesign Program based on Evaluation Identify Problem of Issue Asses Needs & Assets Write Goals Write Measurable Objectives Plan Evaluation Choose & Use Interventions Assess Outcomes Short & Long Term

Avoid Frustration: Ask these Questions Before Beginning Your Program Not After It Has Begun What will you evaluate? What will you evaluate immediately? Less intense skill and knowledge development Pre-Post test or Retrospective Tests What will you evaluate via follow-up assessment? More intense long-term effects, behavior changes, skills and knowledge Surveys or telephone interviews

Key Factors to Planning Keep the “ main thing ” (Objectives and Goals) the ‘ main thing. ’ Is it worthwhile to evaluate a one-shot brief program? “ Two hour rule. ” Evaluate larger efforts, i.e. program series over a year or five years, multi-dimensional efforts to teach, change behavior which lead to measurable IMPACTS. Will follow-up “ Outputs ” be necessary to achieve the desired outcomes? Have you planned for these?

Identification of the Problem or Issue 1.Clear vision of needs and assets of your target  Create a statement page of the problem or issue. 2.Is there  clientele consensus  leader and agency “ buy in ” 3.Facilitate public awareness of problem or issue  Foundation of support 4.Put together statements to create public awareness, seek program and financial support  Contact stakeholders at beginning of program for their support not at the end of the program.

Needs Assessment Who should be assessed or asked about needs? BUGS Formula How will assessment be completed? focus groups interviews written or telephone interviews self-report surveys Review available social and scientific indicators surrounding the issue or problem. Demographic changes, percentages, increases and decreases in…

Writing Goals Long term goals - one to five years to accomplish WHO will DO WHAT for WHAT PURPOSE? Verbs which usually accompany goals include change, promote, offer, increase, decrease, train or enable. Reported Utah bankruptcies in Utah will decrease by 15% in 2000 as a result of the Financial Management, Budgeting Awareness program conducted by Utah State University Extension Micro Level Macro Level Community/neighborhood, region/state - levels Individual, family, ranch, farm, club - levels Goals Written on

Writing Objectives - The Basics Short Term Objectives - One year or less to accomplish They Target: knowledge, attitudes, skills, aspirations, tax dollar support, behaviors and other program characteristics which will change as a result of the intervention. SMART S - Specific M - Measurable A - Attainable R- Relevant T- Time-based The Formula By (DATE) (WHO) will (DO WHAT IMPLEMENTING/ACTION VERB) for (WHAT OUTCOME)?

Evaluation Plan  Overall purpose for the evaluation, and specific questions the evaluation needs to answer. Short summary of what the evaluation is to accomplish  Methods to be used  evaluation strategy  mail survey, telephone survey, etc.  target audience  from which data is collected  time frames  persons responsible  How Results will be used  Budgetary Needs

Model for Evaluation Planning OutcomesSpecific Information Needed- Indicators Data Collection method(s) Date(s) for data collection Data source(s) Program Goals: Integrated Weed Management Program Ecologically based weed management Reduced herbicide use Utilization of IPM Utilization of other weed control methods cultural,biological, GMO ’ c Data searches USDA statistical baseline Dealer Coop records Commodity group data records Nov-Feb statistical collection from groups Measures for pre- herbicidal June On-going IPM measures USDA commodity groups Special interest groups Commercial applicators Scientists involved in specialized research

Plan Evaluation Begin by asking the BIG question What outcomes do I hope to show from this program? Social Environmental Economic Begin with the end in mind! Outcomes and Success Markers We expect to see We would like to see We would love to see Success Markers = Identified actions/behaviors which indicate successful accomplishment of the outcome ? 1.Who: 2.Clause which describes the successful attainment of identified change 3.Complete statement with a behavioral intention that represents a significant attainment for the person targeted Outcome #1 SM1 SM2 SM3

Assess Outcomes Measuring Impacts Techniques to consider include : Interviews Focus Groups Post-test only, post-test then pre-test, pre-test-post-tests, pre-test-post-test follow-up evaluations, experimental-control groups, quasi-experimental or comparison groups Using Measures that determine changes in (KASA) Knowledge Attitude Skills Aspirations Measuring for change in Practices Interventions -That which has changed peoples lives Social Economic Environmental SEE

Reporting Results to Stakeholders Remember who the stakeholders are you previously identified in the initial program planning. REPORTS MAY BE DIFFERENT FOR VARIOUS STAKEHOLDERS. Address the stakeholders ’ needs! - One or two basic demographics about the problem - Your basic objectives/ intention of your program - What you did - What results/outcomes you found. What happened as a result of the program? Generally no longer than one page, concise, organized and clearly presented. Consider an executive summary approach. A Basic Outline to Consider:

Redesign the Program Based on Evaluative Results The most important use of your evaluation results is to use the results to redesign your program so that it will be more effective.ASK: What did you learn? What worked? What did not work? What can you learn about your approach or teaching style? What did you learn about your audience? What additional services should/could be added? Loop back to the beginning phase of program planning and apply the learnings to each step in the planning process. Program Planning Process Redesign Program