1. Determining Program Effectiveness Brad Krevor, Ph.D. Heller Graduate School Brandeis University October 26, 2007

2. Central challenge : How does an agency determine if the programs it has put into place are effective?

3. Measuring effectiveness 1. The wrong way: Assurances of Voluntary Compliance with state Attorneys General: Walgreens: inspect 10% of stores every 6 months

4. Measuring effectiveness Age-verification rates for one chain (as %) Qtr 1Qtr 2Qtr 3Qtr 4Qtr 1Qtr 2Qtr 3Qtr 4 2004 2005 85.470.265.360.672.681.077.452.0 What would you do to improve performance if you were the manager in charge?

5. Measuring effectiveness For that same national chain: Survey of clerks / managers in 450 stores 50% passed both Qtr 1 + Qtr 2, 2004 50% failed Qtr 1 + / or Qtr 2, 2004 Tested 1) knowledge 2) self-reported behavior 3) attitudes and beliefs re: underage sales, and 4) perception of attitudes and beliefs of others

6. Measuring effectiveness 1. What then does this data demonstrate? Qtr 1Qtr 2Qtr 3Qtr 4Qtr 1Qtr 2Qtr 3Qtr 4 2004 2005 85.470.265.360.672.681.077.452.0 -- inadequate data collection design!

7. Measuring effectiveness : a better way

8. Madison pre-post test 5 pre- and post-test inspections at each of 10 randomly selected licensees Inspections Pass FailPass rate 45 pre-test 28 17 62.2% 48 post-test 43 5 89.6% difference = + 27.4% % improvement = 44%

9. Measuring effectiveness Another wrong way: Jacksonville, FL: Report to clerks on ID checking

10. Do Americans support the age 21 drinking age? A scientific survey asked Americans for their opinion: Do you support the current law that makes it illegal to buy alcohol if you are under age 21? 87% of Americans say YES, they support the age 21 drinking age.

11. Do Americans mind waiting in line at the checkout counter while clerks check customer IDs? The survey asked: Tell me if you agree or disagree with the following statement: I do not mind waiting in line at the checkout counter for the additional time it takes clerks to check IDs. 89% agreed that they don’t mind waiting for ID checks

12. Measuring effectiveness Report to Clerks: 2 pre-post mystery shopper tests of treatment and control stores Result: no change in behavior WRONG APPROACH: Intervention was too weak to expect change in behavior.

13. Another approach to effectiveness: Iowa TIPS Training

14. Measuring processes When a measure of effectiveness can not be reasonably expected, processes measures (i.e. measures of activities) may be more appropriate PLCB Report to PA Legislature

15. Measuring processes Caveat: MA Tobacco Control Program

16. Measuring processes Process measures can serve as bench- marks and thus serve as performance measures STOP Act

17. Recommendations Speak with researchers in local colleges / universities about how to evaluate programs Identify evaluators who work with state-funded prevention agencies Bring issues of evaluation and measurement to the NABCA Education Committee Bring issues of evaluation and measurement to the RRF Research Committee