Evaluating Social Marketing Campaigns Emerson College Summer Institute for Social Marketing and Health Communication JuNE 9, 2016
Wotan’s Ravens
Types of Evaluation Formative Process *Delivery/Implementation *Exposure/Reach Outcome (AKA summative or impact)
Important Principles of Evaluation Start thinking about and planning for evaluation from the first day Build it into your budget Look for ways to conduct evaluation economically
Purpose of Formative Evaluation Gathering information to making good choices about audiences, concepts, messages, channels, and materials Formative research becomes the foundation for all important decisions made throughout a social marketing initiative
Formative Evaluation Methods Surveys One-on-one exploratory interviews Intercept interviews Expert reviews Readability testing Focus groups
Cutting Costs with Focus Groups If you need to gather information beyond a single geographic region or gathering people proves a challenge, consider conducting telephone, online, or Webcam groups Get a partner to donate something other than cash that can be used as an incentive
Cutting Costs with Focus Groups (cont.) Send someone from your own team for moderator training Find an inexpensive location to conduct the groups (church, community center, etc.) If taping is needed, contact a local educational institution
Common Mistakes with Formative Evaluation Not enough data Not enough of the right kind of data Using convenience samples to draw firm conclusions
Process Evaluation
Colonel Brighton: “And what are you to do for the Arab Bureau Colonel Brighton: “And what are you to do for the Arab Bureau?” Lawrence: “I’m to appreciate the situation.”
What is Process Evaluation? An extension of formative evaluation A prelude and companion to outcome evaluation Concerned with monitoring and collecting data on the fidelity and implementation of campaign activities
Why Conduct Process Evaluation? Detecting small problems before they become large *Is the social marketing program on track? *Are midcourse course corrections necessary?
Why Conduct Process Evaluation (cont.)? Providing immediate evidence that the system is working *Are we keeping our partners engaged? *Do we need to boost morale?
Why Conduct Process Evaluation (cont.)? Helping with the interpretation of other evaluation results *What context have we provided for further assessment? *What have we learned to help us understand successes and disappointments?
Delivery/Implementation Evaluation
Delivery/Implementation Evaluation
Delivery/Implementation Evaluation Key Questions *Is an effective system in place for communicating a message? *Are activities being carried out as planned? *Is the timetable being followed? *Are the project staff, partners, and volunteers doing their jobs as assigned?
Delivery/Implementation Evaluation (cont.) Problems that delivery/implementation evaluation can uncover *Shortage of resources *Lack of motivation *Poor leadership *Confusion about the implementation plan *Intervening events
Methods of Evaluating Delivery/Implementation Log/tracking systems Systematic checks with implementers, partners, and media
Exposure/Reach Evaluation Key questions *Was the message received as intended? *How many people heard and/or saw the message? *Did the primary and secondary target audiences hear and/or see the message?
Exposure/Reach Evaluation (cont.) Problems that exposure/reach evaluation can uncover *Lack of appropriate assessment of delivery/implementation *Intervening events *Poor planning
Methods of Evaluating Exposure/Reach Monitoring mass media *Monitoring services *Content analysis Materials inventory *Checking stock *Checking distribution points
Methods of Evaluating Exposure/Reach (cont.) Monitoring responses and inquiries *Calls to telephone hotlines *Written requests for materials *Web site hits and inquiries Surveys *Exposure to campaigns *Awareness of campaigns
Limitations of Process Evaluation Data can be misleading Doesn’t tell you how people reacted to a message
Outcome Evaluation
Purpose of Outcome Evaluation Is there evidence that our intervention worked? We need to know for ourselves if the effort was worth it Partners and funders will expect signs of success We need to know how to build on success (or avoid repeating failures)
Grounding Outcome Evaluation in Objectives Outcome evaluation should be guided by objectives that have been refined from the early stages of planning Objectives should be SMART (Specific, Measurable, Achievable, Realistic, and Time-phased)
Examples of SMART Objectives Belief Objective– “By September 2017, increase by 20% the number of middle school students in Barnstable County who believe they are capable of using resistance strategies against drinking at appropriate moments.”
Examples of SMART Objectives Knowledge Objective– “By September 2017, increase by 30% the number of middle school students in Barnstable County who know at least three resistance strategies that can be used when pressured to drink alcohol.”
Examples of SMART Objectives Behavioral Objective– “By September 2017, increase by 10% the number of middle school students in Barnstable County who consistently use resistance strategies when pressured to use alcohol.”
Classic Experiment: Gold Standard for Outcome Evaluation Experimental Group* Pretest Intervention Posttest Control Group* Pretest No Intervention** Posttest *Assumes random assignment (considered quasi-experimental if random assignment doesn’t exist) **Or at least a modified version of the intervention
Not Meeting the Gold Standard: Random Assignment Not Possible Random assignment might not be possible for logistical reasons Try to match to a control group that is similar to the experimental group
Not Meeting the Gold Standard: Comparison Group Not Possible Using a comparison group might be cost prohibitive or logistically impossible due to access A time series design is still very rigorous, but requires several data collections PRPRPRTRPOPOPO
Not Meeting the Gold Standard: Pretest Is Not Possible Intervention might be underway before there is time to collect baseline data or the team can’t afford a pretest Can do a posttest only control group design, but you’ll never know if groups were equivalent at the beginning Might be possible to use secondary data as a baseline
Not Meeting the Gold Standard: Can’t Afford to Collect Any Data Is there a partner that can collect data for you? With increased access to databases online, look for secondary data sources (might have to wait some time for identifying change) Are there reasonable proxy measures to use?