1. WASHINGTON STATE UNIVERSITY EXTENSION Evaluation Based Program Development This material was developed in partnership with Rutgers's University and Lydia B. Blalock, Youth Development Specialist, New Brunswick, New Jersey

2. Why Evaluate? I dunno I think I knew why– once They make me do it –Department chair –Funding organization –My mom I don’t Demonstrate success Understand failure Provide feedback for program improvement –Process –Outcome Nip problems in the bud

3. Process Objectives If I: Develop and present seminar on EBPD Develop and distribute appropriate handouts Engage participants in group activities Provide participants an opportunity to practice Encourage questions THEN

4. Outcome Objectives Participants will: Program Development –Understand how to clearly define issue program will address. –Use program theory to plan activities. –Write clear outcome and process objectives. –Use EBPD Model to plan programs. Program Evaluation –Develop skills to evaluate programs. –Select appropriate evaluation methods and designs. –Select criteria to evaluate objectives. –Use EBPD Model to plan evaluation.

5. BUT FIRST: Is it a Program or a program Stakeholder leadership Faculty leadership Multi-disciplinary Sustainable Academic department Student component Research, teaching, outreach

6. The Steps of Evaluation-Based Program Development

7. Step 1: Define the Problem Describe Current Situation –How does the world look today? Why a Problem –Clearly articulate why a problem –Ask why until get to root of problem –Symptom vs. Cause Identify Stakeholders –Population: Who is affected? How many? –Stakeholder Leadership: interest in solving problem? –Potential Collaborators: Already working on problem? How: Evidence? Learn more? References & Facts –Learn all you can about the issue/problem –

8. Set Priorities –Factors that influence priorities Difference between actual and target states Degree of difficulty in addressing needs Cost of implementing solutions Political and other factors –Derive priorities from several data sources –Priorities without resources Select Solution Strategies to Meet Needs –Consider alternative solutions for each need –Evaluate alternatives –Select solutions to implement –Scholarship Opportunities?

9. Table 1. Define the Problem Identify who, what, where, when, why. 1. Describe Current Situation How does the world look today? 2. Why a Problem? 3. Identify Stakeholders Clearly articulate why current situation is a problem. Ask why until you get to root of problem. Population. Who is affected by problem? How many people are affected? Stakeholder Leadership. Who’s interested in solving problem? Potential Collaborators. Who’s interested in solving problem and/or already working on problem? What is the evidence? How can I learn more? Provide references and facts.

10. TALK IT OVER Fill Out Table 1 with your group.

11. Step 2: Develop Outcome Objectives Describe Ideal Situation –How would world look w/o problem (perfect world)? Define Target Audience/s –Who/what can we change (youth, families, policies, etc.)? –What do we know about culture/s of audience? Identify Outcome Objectives – What changes can I influence in my target audience/s? Identify Resources –What do we need? –What do we have?

12. Step 2. Develop Outcome Objectives 

13. Writing Measurable Objectives Specific –Will increase knowledge vs. Will know 3 sources of Vitamin C Time delineated –At the end of the program, in 6 weeks, etc. Measurable –75%, all, etc. Example –Participants will increase knowledge of nutrition. –75% of program participants will know 3 sources of Vitamin C 6 weeks after the program.

14. Table 2: Clarifying Your Projected Impact

15. Talk it Over Fill out Table 2 with your group. #4: Brainstorm Resources

16. Step 3. Plan Program Use “Program Theory” to develop process objectives! IF:THEN

17. Program Theory: The Key  to Unlocking  Change      

18. Output the Process of an Outcome …wha…? Outcome Objectives: the changes you will influence in your target audience (Learning, Action, Condition) Process Objectives: the events, programs, activities you will conduct, deliver, develop, train, provide, access, facilitate to reach your outcome objectives Program Outputs: your deliverables Outcome & Process Objective Activity

19. Talk it Over Fill out table 3 with your group

20. SO WHAT? Friend to Groucho Marx: “Life is difficult!” Marx to Friend: “Compared to what?”

21. Why We Don’t Evaluate Too hard Not qualified Costs too much Takes too much time Don’t think about it “in time” No one ever reads evaluation reports Everyone “knows” it’s a good program

22. Step 4: Program Evaluation Select Evaluation DesignSelect Data Collection Method

23. Evaluation Design 101 One-Shot –Data collected following program, Retrospective Pretest –Data collected following program; participants recall behavior prior to and following program. One Group Pretest-Posttest - Data collected prior to and following program Time Series –Data collected prior to, during, after program Pretest-Posttest Control-Group –Data collected from 2 different groups prior to and following program. –Treatment group: program –Control group: no program Post-Only Control-Group –Data from 2 different groups collected following program only

24. Evaluation Design 101 Evaluation DesignPretest Posttes t # Times Assessed Follow-up Control Group One-Shot Design X1 Retrospective Pretest XX1 One Group Pretest-Posttest XX2 Time Series XX3 (min)X Pretest-Posttest Control- Group XX 2x per group X Post-Only Control-Group X 1x per group X

25. Data Collection Methods Face to Face Surveys –Respondent completes survey in presence of evaluator –Evaluator may record responses and is able to answer questions Self Administered Questionnaires –Respondent given survey to complete on his/her own. –Respondent’s responsibility to return survey. Personal Interviews –Respondent asked pre-determined questions. –Evaluator records responses, can ask respondent to clarify answers. Focus Groups –Several respondents meet with evaluator. –Evaluator facilitates discussion around pre-determined questions. –Discussion recorded.

26. Learned Anything? Let’s Check!

27. Learning, Action & Condition Objectives

28. Think it Over Fill out Table 4 with your group. Learning Indicators Action Indicators Condition Indicators

29. Cultural Aspects of EBPD What is culture? –“Behaviors and beliefs characteristic of a specific social, ethnic, or age group.” –We are a product of our culture –Culture influences (unconsciously) what we think of people from other cultures. –Recognize cultural biases towards participants and strive to eliminate them. Be aware of how differences between cultures may impact EBPD Neutralize cultural biases and prevent EBPD design flaws: –Know your audience (Step 2 – Develop Outcome Objects) –Develop cross-cultural efficacy!

30. Culturally Sensitive Design Cultural Competence vs. Cross-cultural Efficacy Cultural competence I can deal with them. My culture is superior to their culture. (It’s all about me!) Cross-cultural efficacy I can effectively interact with people from other cultures. I can benefit from our interactions. Neither culture is superior to the other. (It’s all about us!)

31. Reporting & Scholarship Table 6. Dissemination and Scholarship  How will you share your work? Evaluation-Based Program Development 1. Identify Scholarship Activities 2. Identify Products & Deliverables Table 1. Define Problem Table 2. Identify Outcome Objectives and Resources Table 3. Plan Program Table 4. Plan Program Evaluation Table 5. Implement Program, Conduct Evaluation and Document Impact Table 6. Dissemination and Scholarship Report Results Demonstrate Impact on Profession Attain Recognition

33. Process Objectives If I, as facilitator: Develop and present seminar on EBPD Develop and distribute appropriate handouts Engage participants in group activities Provide participants an opportunity to practice Encourage questions THEN

34. Outcome Objectives Participants will: Program Development –Understand how to clearly define issue program will address. –Use program theory to plan activities. –Write clear outcome and process objectives. –Use EBPD Model to plan programs. Program Evaluation –Develop skills to evaluate programs. –Select appropriate evaluation methods and designs. –Select criteria to evaluate objectives. –Use EBPD Model to plan evaluation.

35. Condition Changes We’ll report stunning impacts from well planned programs And the folks in Pullman will love us the most And people will dump bucket loads of money into our programs And the world will be a better place!