1. Approaches for Evaluating Local Environmental Efforts
Massachusetts Statewide Substance Abuse Prevention Conference
At the Educational Development Center
October 30, 2014 – Waltham, MA

2. The Wayne Harding Slide
Massachusetts
Wyoming
November 15, 2018

3. Environmental Mind Experiment

4. Challenges Who do we count? How do we know things are changing?
Processes
Outcomes
Who do we count?
When do we count them?
For how long do we count them?
What else do we count?
How do we know things are changing?
What caused the change?
Did we reach our outcomes?
November 15, 2018

5. Discussion: What are some of the challenges you have
faced when trying to evaluate
environmental prevention
Strategies?
November 15, 2018

6. Tectonic Shift in Thinking
Serving People
Changing Communities
November 15, 2018

7. New Goals and Objectives
Provide programs to bring about individual changes, or …
… create changes in the system to bring about population-level change
November 15, 2018

8. New Unit of Analysis
From Individuals
To Populations
November 15, 2018

9. New Discipline
Program Evaluation
Policy Analysis
November 15, 2018

10. Process Evaluation
November 15, 2018

11. Outcome Evaluation Models versus Individual Variables Ingredients
Dessert
November 15, 2018

12. Outcome Evaluation Designs
Interrupted Time Series Analysis
November 15, 2018

13. Outcome Evaluation Designs
Control Series Analysis
November 15, 2018

14. Outcome Evaluation Designs
Interrupted Time Series Analysis with Forecast
November 15, 2018

15. Choosing Comparison Communities
Demographics
Educational Systems
Economic Base
Not receiving the intervention
Common Baseline Data
Example: School Random Drug Testing Policy
November 15, 2018

16. Choosing Comparison Communities
November 15, 2018

17. Questions & Answers

18. Question: What is the best coffee shop within walking distance of EDC
Question: What is the best coffee shop within walking distance of EDC? (and don’t say Dunkin Donuts)

19. Utah Parents Empowered
Evaluating Media
Utah Parents Empowered
Media Campaign
November 15, 2018

20. Evaluation Starts with a Question
November 15, 2018

21. Awareness? Enrollment? Knowledge? Attitudes? Behavior?
1. Ask Questions
Awareness?
Enrollment?
Knowledge?
Attitudes?
Behavior?
November 15, 2018

22. 2. Decide if its Whole or Stand Alone?
November 15, 2018

23. 3. Collect and Analyze Data
Quantitative
Qualitative
Archives
Observation
Surveys
Archives
Observation
Interviews
Focus Groups
November 15, 2018

24. Evaluating Media Example
Perkins, W.H., Linkenbach, J.W., Lewis, M.A., & Neighbors, C. (2010). Effectiveness of social norms media marketing in reducing drinking and driving: A statewide campaign. Addictive Behaviors, 35,
November 15, 2018

25. Evaluating Media Example
Television
Radio
Local Newspaper
College Newspaper
Promotional Items
Theater Ads
Billboards
Indoor Ads
January 2002 through March 2003
November 15, 2018

26. Target population of 21 to 34 year-olds surveyed four times
Evaluating Media Example
Target population of 21 to 34 year-olds surveyed four times
n=1669
n=765
n=1047
November 15, 2018

27. Evaluating Media Example
Differences for Intervention and Comparison Counties
between November 2001 and June 2003
November 15, 2018

28. Questions & Answers

29. Discussion: Let’s do this together Who has a media campaign
that needs evaluating?
November 15, 2018

30. Consider the media campaign goals and evaluation research question
Step One
Consider the media campaign goals and evaluation research question
What are you trying to change?
Is the media part of a larger effort or does it stand alone?
November 15, 2018

31. Decide on a methodology/
Step Two
Decide on a methodology/
research design
Can you find a comparison community?
How much time do you have?
November 15, 2018

32. Identify indicators or create data collection instruments
Step Three
Identify indicators or create data collection instruments
How will you tell the story (processes)?
How will you measure change (outcomes)?
November 15, 2018

33. Step Four Improve the Media Create a Report Make Decisions
Create a Presentation
November 15, 2018

34. Questions & Answers

35. Final Question: Just to be clear, the best nearby coffee shop is?

36. Contact Me! Slides available at… rodneywambeam.com presentation tab or
(307)
rodneywambeam.com
Slides available at…
rodneywambeam.com
presentation tab or
from MassTAPP
November 15, 2018

37. Thank You!
November 15, 2018