Amy Livingston

 Process evaluation focuses on assessing and documenting grantees’ : ◦ Partnerships and capacity building ◦ Assessment and planning ◦ Implementation of strategies (including successes and challenges encountered, and program-level outcomes)

 Outcome evaluation focuses on: ◦ Assessing change over time in population-based outcome measures and targeted intervening variables ◦ Comparing change in outcomes in grantee communities (both individually and collectively) to the rest of the state ◦ Primary sources will be YRBS and YAS

Formative Evaluation Provides template for implementation activity Helps identify implementation problems/issues that may be correctable Documentation/replicability Provides record of what was done Performance monitoring Provides metrics for assessing progress in achieving implementation objectives and goals Interpretation of program outcomes Helps explain why or why not outcomes were achieved

 Most of our process data for PFS comes from data submitted quarterly to CGRS  Also site visits, networking calls, meetings with District Office staff  May do some qualitative evaluation (focused interviews and/or focus groups) to help assess changes in regional capacity

 All grantees have reported an overall increase in collaboration with community partners since the beginning of the grant

Period 10/13 thru 3/144/14 thru 9/14 Sector: Number of Grantees with at Least One Partner Number of Grantees with At Least One Partner Collaborating at High or Very High Level Number of Grantees with at Least One Partner Number of Grantees with At Least One Partner Collaborating at High or Very High Level Business community 5466 Civic/volunteer group 3352 Clergy/religious organization 3153 Healthcare professionals 6364 Judiciary 2130 Law enforcement agency 6565 Media 6463 Parent 5353 School 6554 State, local and/or tribal governmental agency 6465 Youth 6363 Youth serving organization 6464 Other individual or organization involved in preventing or reducing substance abuse 4363 Colleges 5554 Young adults 5252 Number of Grantees with Partners in Each Sector and with Partners Collaborating at a High Level

 From the beginning of the grant through the report submitted in July 2014, TA received was assessed by intervention and by source (PC, other ADAP staff, etc.)  Starting with the report submitted in October, TA is now being assessed overall by topic area (SPF step) and source  Will be used by ADAP to monitor and improve quality of TA

 CGRS includes a measure of progress on each of the key activities from the work plan for each evidence-based intervention  Grantees are asked to rate progress on each key activity as not started, partially completed, in progress, or completed  In addition, there are other process measures in CGRS that collect both quantitative and qualitative information for each intervention

Intervention: Number of grantees implementing intervention Started on at least half of the steps Started on all of the steps Completed all of the steps Started but not able to complete 1 or more steps Restrictions on alcohol outlet density Restrictions on alcohol in public places Enhancements to social host liability Enhancements to open container regulations Restrictions on advertising and promotion Saturation/Party Patrols Community mobilization Media advocacy e-CHECKUP for College Alcohol Edu Guiding Good Choices Nurturing Parenting Program Parenting Wisely e-CHECKUP for the Community Progress on Activity Implementation, by Intervention, April 2014 through September 2014

Type of policy educationNumber of grantees implementing Number of policies developed Number of policies approved Enhancements to open container regulations 111 Enhancements to restrictions on advertising and promotion 100 Enhancements to social host liability 100 Restricting alcohol outlet density 211 Restrictions on alcohol in public places and/or at community events 500

 Successes: ◦ Two successful town-level policy changes so far ◦ Getting health language added to town or regional plans ◦ Systematic review and assessment of current policies and ordinances ◦ Development of summary document that can be shared with towns throughout the region  Challenges: ◦ Lack of stakeholder motivation ◦ Lack of existing model policies ◦ Takes time!

 All 6 grantees are working with law enforcement to enhance saturation and/or party patrols (work plans and reporting on these two interventions were combined in July) Successes:  Getting MOUs in place  Establishing relationships with law enforcement Challenges:  Coordination with different law enforcement agencies  Low capacity to conduct additional patrols/utilize overtime Law enforcement activityTotal number Proactive patrols conducted49 Minor in possession citations issued69 Parties responded to by patrols21

Type of InterventionNumber of Grantees Implementing Total number of individuals reached E-Checkup for College22 Alcohol Edu1342 E-Checkup for the Community 274 Successes:  High completion rate of Alcohol Edu at JSC  Decrease in alcohol policy violations at JSC; from 48 in fall of 2012 to 14 in fall of 2013 Challenges:  Getting colleges to commit to using e-Checkup

Type of programNumber of grantees implementing Total number of parents reached Guiding Good Choices146 Nurturing Parenting Program (Nurturing Fathers) 116 Parenting Wisely111 Successes:  Centralized coordination of parenting classes around the county worked well  Positive feedback from participants Challenges:  Getting parents to sign up and follow through with sessions

ActivityNumber of grantees Relevant data Retailer recognition for passing compliance checks 6 A total of 123 businesses were recognized Support activities for responsible beverage service training 6 Grantees assisted with a total of 19 Merchant Education Seminars Sticker Shock5 A total of 42 businesses participated A total of 63 youth volunteers participated

 A total of 3 new safe drop-off locations were established Number of PFS regions using different types of outreach to educate the community about proper storage and safe disposal of unused prescription drugs. Type of outreachNumber of grantees using Brochures4 Community events6 Print media4 Press release3 PSA2 Radio3 Social media6 Television2 Website5 Other (e.g. flyers, posters, newsletters, Rx bag inserts) 4

 Successes: ◦ Creative ways to get the word out e.g. through senior centers, Meals on Wheels ◦ Hungry Heart screenings and opiate summits have provided good forum for communicating concrete things community members can do ◦ Great collaboration with law enforcement and health care providers on this issue ◦ Collaboration with health care providers and pharmacists has led to new understanding of the issue for all involved and allows for greater saturation of messages  Challenges: ◦ Waiting for statewide materials ◦ Engaging pharmacists, especially at chain pharmacies

 Successes: ◦ Youth engagement efforts very strong in some regions ◦ Increased collaboration with schools ◦ Community forums on opiate use and Hungry Heart screenings provided great opportunities  Challenges: ◦ Partners and collaborating agencies don’t always identify with county-wide approach ◦ How to engage with culturally diverse sub-groups

 Grantees contacted the media a total of 185 times  The total number of instances of earned media reported across all PFS regions was 488 (*true number is probably closer to 76) Number of instances of earned media by type *In reviewing the description of types of media included in the “other” category, it was clear that some of the instances in this total would not be considered earned media but rather grantee- generated media such as ads, flyers, press releases, etc. Type of mediaNumber of instances Letter to the editor 3 Newspaper story 43 Editorial/op-ed 6 TV story 16 Radio interview or story 6 Blog story/discussion 2 Other 402*

Number of times each type of message was addressed by earned media *The description of other types of messages included educational opportunities to prevent substance abuse, and youth engagement (specifically an Above the Influence retreat) ^These items contribute to the ADAP media advocacy performance measure on the Dashboard Type of messageNumber of instances Community efforts to reduce UAD 38 Enforcement efforts to reduce UAD 11 Community efforts to reduce Rx misuse^ 68 Community specific data or info on UAD 15 Community specific data or info on Rx misuse^ 29 General info or data related to UAD (not community specific) 21 General info or data related to Rx misuse (not community specific)^ 53 Other* 329

 Successes: ◦ Positive relationships developed with local media ◦ Good coverage of Take Back Days and summits on opiate use ◦ Invitations to be interviewed for TV or papers ◦ Invitations for regular media spot on TV or newspaper column  Challenges: ◦ Developing process for coordinated media outreach and tracking at regional level ◦ Ability/capacity to monitor local media

 PIRE is preparing a year one evaluation brief for ADAP which is primarily a summary of CGRS data collected so far.  We are also working with ADAP to develop performance measures related to the PFS, some of which may be displayed on the Dashboard  Modify CGRS when CLI goes live