Download presentation
Presentation is loading. Please wait.
1
Evaluating Social Marketing Campaigns
Emerson College Summer Institute for Social Marketing and Health Communication JuNE 9, 2016
2
Wotan’s Ravens
4
Types of Evaluation Formative Process *Delivery/Implementation
*Exposure/Reach Outcome (AKA summative or impact)
5
Important Principles of Evaluation
Start thinking about and planning for evaluation from the first day Build it into your budget Look for ways to conduct evaluation economically
6
Purpose of Formative Evaluation
Gathering information to making good choices about audiences, concepts, messages, channels, and materials Formative research becomes the foundation for all important decisions made throughout a social marketing initiative
7
Formative Evaluation Methods
Surveys One-on-one exploratory interviews Intercept interviews Expert reviews Readability testing Focus groups
8
Cutting Costs with Focus Groups
If you need to gather information beyond a single geographic region or gathering people proves a challenge, consider conducting telephone, online, or Webcam groups Get a partner to donate something other than cash that can be used as an incentive
9
Cutting Costs with Focus Groups (cont.)
Send someone from your own team for moderator training Find an inexpensive location to conduct the groups (church, community center, etc.) If taping is needed, contact a local educational institution
10
Common Mistakes with Formative Evaluation
Not enough data Not enough of the right kind of data Using convenience samples to draw firm conclusions
11
Process Evaluation
12
Colonel Brighton: “And what are you to do for the Arab Bureau
Colonel Brighton: “And what are you to do for the Arab Bureau?” Lawrence: “I’m to appreciate the situation.”
13
What is Process Evaluation?
An extension of formative evaluation A prelude and companion to outcome evaluation Concerned with monitoring and collecting data on the fidelity and implementation of campaign activities
14
Why Conduct Process Evaluation?
Detecting small problems before they become large *Is the social marketing program on track? *Are midcourse course corrections necessary?
15
Why Conduct Process Evaluation (cont.)?
Providing immediate evidence that the system is working *Are we keeping our partners engaged? *Do we need to boost morale?
16
Why Conduct Process Evaluation (cont.)?
Helping with the interpretation of other evaluation results *What context have we provided for further assessment? *What have we learned to help us understand successes and disappointments?
17
Delivery/Implementation Evaluation
18
Delivery/Implementation Evaluation
19
Delivery/Implementation Evaluation
Key Questions *Is an effective system in place for communicating a message? *Are activities being carried out as planned? *Is the timetable being followed? *Are the project staff, partners, and volunteers doing their jobs as assigned?
20
Delivery/Implementation Evaluation (cont.)
Problems that delivery/implementation evaluation can uncover *Shortage of resources *Lack of motivation *Poor leadership *Confusion about the implementation plan *Intervening events
21
Methods of Evaluating Delivery/Implementation
Log/tracking systems Systematic checks with implementers, partners, and media
22
Exposure/Reach Evaluation
Key questions *Was the message received as intended? *How many people heard and/or saw the message? *Did the primary and secondary target audiences hear and/or see the message?
23
Exposure/Reach Evaluation (cont.)
Problems that exposure/reach evaluation can uncover *Lack of appropriate assessment of delivery/implementation *Intervening events *Poor planning
24
Methods of Evaluating Exposure/Reach
Monitoring mass media *Monitoring services *Content analysis Materials inventory *Checking stock *Checking distribution points
25
Methods of Evaluating Exposure/Reach (cont.)
Monitoring responses and inquiries *Calls to telephone hotlines *Written requests for materials *Web site hits and inquiries Surveys *Exposure to campaigns *Awareness of campaigns
26
Limitations of Process Evaluation
Data can be misleading Doesn’t tell you how people reacted to a message
27
Outcome Evaluation
28
Purpose of Outcome Evaluation
Is there evidence that our intervention worked? We need to know for ourselves if the effort was worth it Partners and funders will expect signs of success We need to know how to build on success (or avoid repeating failures)
29
Grounding Outcome Evaluation in Objectives
Outcome evaluation should be guided by objectives that have been refined from the early stages of planning Objectives should be SMART (Specific, Measurable, Achievable, Realistic, and Time-phased)
30
Examples of SMART Objectives
Belief Objective– “By September 2017, increase by 20% the number of middle school students in Barnstable County who believe they are capable of using resistance strategies against drinking at appropriate moments.”
31
Examples of SMART Objectives
Knowledge Objective– “By September 2017, increase by 30% the number of middle school students in Barnstable County who know at least three resistance strategies that can be used when pressured to drink alcohol.”
32
Examples of SMART Objectives
Behavioral Objective– “By September 2017, increase by 10% the number of middle school students in Barnstable County who consistently use resistance strategies when pressured to use alcohol.”
33
Classic Experiment: Gold Standard for Outcome Evaluation
Experimental Group* Pretest Intervention Posttest Control Group* Pretest No Intervention** Posttest *Assumes random assignment (considered quasi-experimental if random assignment doesn’t exist) **Or at least a modified version of the intervention
34
Not Meeting the Gold Standard: Random Assignment Not Possible
Random assignment might not be possible for logistical reasons Try to match to a control group that is similar to the experimental group
35
Not Meeting the Gold Standard: Comparison Group Not Possible
Using a comparison group might be cost prohibitive or logistically impossible due to access A time series design is still very rigorous, but requires several data collections PRPRPRTRPOPOPO
36
Not Meeting the Gold Standard: Pretest Is Not Possible
Intervention might be underway before there is time to collect baseline data or the team can’t afford a pretest Can do a posttest only control group design, but you’ll never know if groups were equivalent at the beginning Might be possible to use secondary data as a baseline
37
Not Meeting the Gold Standard: Can’t Afford to Collect Any Data
Is there a partner that can collect data for you? With increased access to databases online, look for secondary data sources (might have to wait some time for identifying change) Are there reasonable proxy measures to use?
Similar presentations
