Northeast Regional Expert Team Education Development Center, Inc. 55 Chapel Street Newton, MA EDC-CAPT CSAP’s Center for the Application of Prevention Technologies Tools and Tales: Measuring the Success of Environmental Strategies Chuck Klevgaard, Regional Coordinator Pamela Imm, CAPT Associate Erica Schmitz, CAPT Associate CADCA’s 20 th Annual National Leadership Forum February 8-11, 2010 National Harbor, MD

CSAP’s Center for the Application of Prevention Technologies Objectives 1.To understand recent advances in the field associated with the selection, implementation and evaluation of environmental strategies. 2.To understand specific evaluation methods, strategies and tools that can be used to measure short- and long-term outcomes applicable to particular environmental strategies. 3.To understand how communities can successfully implement environmental strategies that can be linked to community- and state-level outcomes.

CSAP’s Center for the Application of Prevention Technologies The CAPT’s Role Builds the capacity of state TA and training systems to select and implement environmental strategies at the local level Provides trainings to local state grantees on the selection, implementation and evaluation of environmental strategies Provides networking opportunities for grantees implementing environmental strategies across states.

CSAP’s Center for the Application of Prevention Technologies What is Getting To Outcomes (GTO)? GTO is a system based on 10 accountability questions that include elements of successful planning, implementation, and evaluation Manuals published by the RAND Corporation Primary Authors: Pam Imm, Matthew Chinman, Abe Wandersman

CSAP’s Center for the Application of Prevention Technologies Background of Getting to Outcomes Initially published by CSAP in 1999 for the Drug Free Community Grantees Updated versions including 2007: Getting to Outcomes for Underage Drinking and other contents are available (e.g., pregnancy prevention, youth development) American Evaluation Association Outstanding Publication

CSAP’s Center for the Application of Prevention Technologies Accountability Questions® What are the underlying needs and conditions that must be addressed? (NEEDS/RESOURCES) What are the goals, target populations, and desired outcomes? (GOALS) What evidence-based programs/practices can be useful in achieving the goals? (PRACTICES)

CSAP’s Center for the Application of Prevention Technologies Accountability Questions® (continued) How do these programs/practices fit with existing initiatives? (FIT) What capacities are needed to implement these selected programs/practices? (CAPACITIES) What is the plan for implementing these programs/practices? (PLAN)

CSAP’s Center for the Application of Prevention Technologies Accountability Questions® (continued) How will the quality of the implementation be assessed? (PROCESS EVALUATION) How well is it working? (OUTCOME EVALUATION) How will continuous quality improvement strategies be incorporated? (CQI) If successful, how will the programs/outcomes be sustained? (SUSTAIN)

CSAP’s Center for the Application of Prevention Technologies Getting to Outcomes Circle Chart Best Practices Fit Capacity Plan Process Evaluation Outcome Evaluation ImproveSustainConditionsGoals

CSAP’s Center for the Application of Prevention Technologies Who Uses Getting To Outcomes? Community coalitions State level agencies Program directors Funders to promote accountability Viewed as a way to operationalize the Strategic Prevention Framework (SPF)

CSAP’s Center for the Application of Prevention Technologies Evidence-Based Environmental Strategies Compliance Checks Alcohol Outlet Density Responsible Beverage Service Social Host Laws Sobriety Checkpoints/Traffic Safety Checkpoints

CSAP’s Center for the Application of Prevention Technologies Evidence-Based Environmental Strategies (contd.) Happy Hour Restrictions Restricting Alcohol Availability at Events Increasing Taxes on Alcohol Graduated Driver’s Licensing Laws Keg Registration

CSAP’s Center for the Application of Prevention Technologies Accessing Getting To Outcomes RAND Corporation (General ATOD prevention) (underage drinking prevention (youth development)

CSAP’s Center for the Application of Prevention Technologies Population-Level Data Maine Youth Drug and Alcohol Use Survey (MYDAUS) Portland Parent Phone Survey Law Enforcement Violations Complemented by 1x1, focus groups, anecdotal

Assessment/Baseline: Youth Perceptions Alcohol is easy to get Won’t get caught by police Won’t get caught by parents Source: Grades 6-12, Portland Public Schools, Maine Youth Drug and Alcohol Use Survey, 2004

Assessment/Baseline: Parent Perceptions Teen drinking is inevitable. Dangers: legal trouble, drinking/driving. I can’t influence teen’s choices about alcohol. I don’t know what actions to take. Better to teach my kid to drink responsibly. Supervise & take the keys. Source: Portland Parent Phone Survey, 2006

Assessment/Baseline: Enforcement 12 underage drinking violations per year 3 furnishing/hosting violations per year No liquor enforcement with retailers Source: Portland Police Department, 2004 Violations

CSAP’s Center for the Application of Prevention Technologies Planning/Strategic Priorities Strengthen enforcement (& youth perception that they’ll be caught) Strengthen parental monitoring (& youth perception that they’ll be caught) Reduce availability (& youth perception that it’s easy to get)

CSAP’s Center for the Application of Prevention Technologies Implementation: Multi-Strategy Approach Collaboration/Outreach Policy (PD, DA, retailers, schools) Training (officers, retailers, seller/servers) Enforcement Media advocacy Marketing, information dissemination FMI:

CSAP’s Center for the Application of Prevention Technologies Outcome Evaluation Did we strengthen enforcement? … And youth perception that they’ll be caught?

Source: Portland Police Department Results from Enforcement Data Underage Drinking Violations

Results: Student Survey Data “I would be caught by police” Source: 2004, 2006, 2008 MYDAUS, Grades 6-12

CSAP’s Center for the Application of Prevention Technologies Outcome Evaluation Did we reduce availability? …And youth perception that it’s easy to get?

Source: Portland Police Department Results From Enforcement Data Furnishing and Hosting Violations

Enforcement Data: Retail Compliance Checks 2007: 170 violations, 74% compliance 2008: 61 violations, 91% compliance 2009: 18 violations, 99% compliance Source: Portland Police Department

RESULTS: Student Survey Data Availability: “Alcohol is hard or very hard to get” Source: 2004, 2006, 2008 MYDAUS, grades 11 and 12

CSAP’s Center for the Application of Prevention Technologies Outcome Evaluation Did we strengthen parental monitoring? …And youth perception that they’ll be caught?

RESULTS: Parent Survey, Attitudes 40% increase in AGREE: “I have the ability to influence my teen’s decisions about alcohol.” 130% increase in STRONGLY DISAGREE: “My job isn’t to prevent my teen from drinking. Rather it’s to teach my teen to drink responsibly.” Source: Parent Survey,

RESULTS: Parent Survey Concern Regarding Long-Term Effects Source: Parent Survey,

RESULTS: Parent Survey Prevention Behaviors: What Actions Do You Take Source: Parent Survey,

RESULTS: Parental Monitoring “I would be caught by my parents” Source: 2004, 2006, 2008 MYDAUS, Grades 6-12

Outcome Evaluation Did we reduce underage drinking?

Source: 2004, 2006, 2008 Portland MYDAUS, Grades 6-12 RESULTS: Student Survey Data 30-Day Alcohol Use

Source: 2004, 2006, 2008 Portland MYDAUS, Grades 6-12 RESULTS: Student Survey Data Two Week Binge Drinking

CSAP’s Center for the Application of Prevention Technologies Portland vs. Statewide Trends SPF-SIG Funding focused on similar strategies across Maine Implementation started 2006; 2-year delay compared to Portland Maine data shows similar trend 2-year lag compared to Portland

CSAP’s Center for the Application of Prevention Technologies For More Information  Maine's Environmental Substance Abuse Prevention Center (MESAP) Training, Consulting, Technical Assistance - Erica Schmitz  21 Reasons Data & Reports  Maine Evaluation Reports