Determining Program Effectiveness Brad Krevor, Ph.D. Heller Graduate School Brandeis University October 26, 2007

Central challenge : How does an agency determine if the programs it has put into place are effective?

Measuring effectiveness 1. The wrong way: Assurances of Voluntary Compliance with state Attorneys General: Walgreens: inspect 10% of stores every 6 months

Measuring effectiveness Age-verification rates for one chain (as %) Qtr 1Qtr 2Qtr 3Qtr 4Qtr 1Qtr 2Qtr 3Qtr What would you do to improve performance if you were the manager in charge?

Measuring effectiveness For that same national chain: Survey of clerks / managers in 450 stores 50% passed both Qtr 1 + Qtr 2, % failed Qtr 1 + / or Qtr 2, 2004 Tested 1) knowledge 2) self-reported behavior 3) attitudes and beliefs re: underage sales, and 4) perception of attitudes and beliefs of others

Measuring effectiveness 1. What then does this data demonstrate? Qtr 1Qtr 2Qtr 3Qtr 4Qtr 1Qtr 2Qtr 3Qtr inadequate data collection design!

Measuring effectiveness : a better way

Madison pre-post test 5 pre- and post-test inspections at each of 10 randomly selected licensees Inspections Pass FailPass rate 45 pre-test % 48 post-test % difference = % % improvement = 44%

Measuring effectiveness Another wrong way: Jacksonville, FL: Report to clerks on ID checking

Do Americans support the age 21 drinking age? A scientific survey asked Americans for their opinion: Do you support the current law that makes it illegal to buy alcohol if you are under age 21? 87% of Americans say YES, they support the age 21 drinking age.

Do Americans mind waiting in line at the checkout counter while clerks check customer IDs? The survey asked: Tell me if you agree or disagree with the following statement: I do not mind waiting in line at the checkout counter for the additional time it takes clerks to check IDs. 89% agreed that they don’t mind waiting for ID checks

Measuring effectiveness Report to Clerks: 2 pre-post mystery shopper tests of treatment and control stores Result: no change in behavior WRONG APPROACH: Intervention was too weak to expect change in behavior.

Another approach to effectiveness: Iowa TIPS Training

Measuring processes When a measure of effectiveness can not be reasonably expected, processes measures (i.e. measures of activities) may be more appropriate PLCB Report to PA Legislature

Measuring processes Caveat: MA Tobacco Control Program

Measuring processes Process measures can serve as bench- marks and thus serve as performance measures STOP Act

Recommendations Speak with researchers in local colleges / universities about how to evaluate programs Identify evaluators who work with state-funded prevention agencies Bring issues of evaluation and measurement to the NABCA Education Committee Bring issues of evaluation and measurement to the RRF Research Committee