Stefanie Murray, MPHPatrick J. Lemmon Public Health Prevention Specialist, CDCCo-founder and former Office of Family Health, OHAExecutive Director Men Can Stop Rape

Learning Objectives By the end of this session, participants will: Understand how to plan a simple program evaluation Feel confident that they can conduct a basic process evaluation of their programs Feel confident that they can conduct a basic outcome evaluation of their programs

Text Chat Question When you think of doing evaluation, what words or feelings come to mind?

What Is Evaluation? Evaluation is a systematic process to understand what a program does and how well the program does it.

Types of Evaluation Formative (before or during activities) Process Evaluation Needs Assessment Summative (after activities are completed) Outcome Evaluation Impact Evaluation

Today’s Discussion Our mission is to: 1. help programs to develop indicators that will help them to determine whether their programs are working as they hope 2. share those results with other programs who are trying to do the same KEEP IT SIMPLE

A Few Key Terms Triangulation: The idea that multiple indicators of success will be more instructive than only one Can look at specific program components Allows for easier mid-course corrections Less likely to show bias Each component is lower stakes Indicator: Measure of success

Stages of Evaluation 1. Planning 2. Program Implementation 3. Data Collection 4. Analysis *Opportunities for mid-course correction to program 5. Reporting 6. Program Review and Revision

Planning: Key Questions What are we hoping will change as a result of the program? How will we know that the change has happened? How can we measure the changes? What are the key components of implementation?

A Note on Measures There are many tools you can use for this Do not rely only on surveys Think creatively, and incorporate into programming if you can Examples of tools: Interviews Pictures of activities Focus groups Facilitator notes

Program Implementation: Key Questions Are we on schedule with our implementation? Any unanticipated barriers or changes?

Data Collection: Key Questions How closely did our implementation follow the plan? Do we have all of the tools we need for the data collection?

Analysis: Key Questions What changes have we seen as a result of our program? Is the change as large as we anticipated it would be?

Reporting: Key Questions Who needs to know this, and what is the best way to share it with them? Newsletter Webinar Funder report

Program Review and Revision: Key Questions How did our implementation look, compared to our plan? Do we need to modify the plan? Did we get the changes we wanted? If not, can we make tweaks to increase the likelihood that we will do better next time?

Planning Our plan is to develop a coalition to convince a local school board to make changes to their policies related to sexual violence prevention programming At least 5 community partners Invited to speak before school board at least twice At least 3 of our proposed changes will be implemented

Program Implementation Track attendance at the coalition meetings and school board hearings Note which organizations take on tasks, since this is an indicator of their buy-in

Data Collection Participation rates at community meetings and school board meetings Have a year-end party for the participants at which we do an informal focus group Did you feel like part of a united effort? Did you feel you contributed actively? Did we make a difference? How? Note how often school board members reference members of coalition in their remarks Count our proposed changes that are implemented

Analysis Did we get at least 5 active partners in the coalition? How long did that take? Did partners feel like part of an effort? Did we get invited to testify? How many times? How many of our proposals were implemented?

Program Review and Revision What were the critiques we received from the partners in the focus group? How could they have felt more engaged? Can we make these partnerships last and take on additional efforts?

Planning While developing curriculum, ask “How will we know we got there?” Example: Bystander intervention session

Decide on tactics We’ll hand out 3x5 cards to participants and ask them to write how they would respond to this scenario: You’re with a group of your friends (all male), and one of them starts joking about a female classmate, who you know but not very well. He calls her an “uptight bitch.” A few of your friends laugh, but others are quiet and look a little uncomfortable. What do you do?

Decide on criteria Can’t just decide after-the-fact what the standards are. We are hoping for answers that are: Nonviolent Creative Recognize humanity of target and of person being intervened with Encouraging more pro-social behavior

Decide what success looks like It is unlikely that we will get to 100% with our efforts, or else our test is probably not a very good one We will divide answers into GREAT, DECENT, and UNDESIRABLE We want at least 60% of our students to give at least a “decent” answer

Program implementation

Data collection and Analysis How would you rate the following responses? I wouldn’t do anything. I support my friend. U

How would you rate the following responses? I’d tell my friend to show some respect. D Data collection and Analysis

How would you rate the following responses? I’d wait until later and then pull my friend aside and tell him that I wish he would be more respectful of other people, both the girl he was talking about and his friends who were uncomfortable with what he said. G Data collection and Analysis

How would you rate the following responses? I’d tell him to shut the hell up. D Data collection and Analysis

How would you rate the following responses? I wouldn’t laugh or anything, but I would not challenge my friend. U Data collection and Analysis

How would you rate the following responses? I’d ask him how he would feel if someone talked about his sister that way. G Data collection and Analysis

How would you rate the following responses? I’d remind him that he got into a fight last year when he heard someone saying something like that about his girlfriend. G Data collection and Analysis

How would you rate the following responses? I’d look at him so he’d know I didn’t like what I said, but I would not say anything. D

How would you rate the following responses? I’d laugh. U Data collection and Analysis

How would you rate the following responses? Something almost like this happened to me last week. I told my friend that I didn’t like what he said, and he kind of laughed it off, but then later he came up to me and apologized for offending me. GG Data collection and Analysis

Sort results GREATDECENTUNDESIRABLE 34 3

Compare to goal GOALRESULTS 67

Reporting “WE ARE AWESOME !”

Questions?

THANK YOU! Stefanie MurrayPatrick Lemmon