Quarterly Meeting June 4, 2019 Al Stein-Seroussi State of South Carolina Department of Alcohol and Other Drug Abuse Services Partnerships for Success - ECHO Project Outcome and Impact Evaluation Quarterly Meeting June 4, 2019 Al Stein-Seroussi

To review concepts related to outcome and impact evaluations Purpose of the Session To review concepts related to outcome and impact evaluations To review MIS spreadsheets in related to outcome and impact evaluations (Not discussing outputs)

SC ECHO Sites

ECHO Coalitions, Counties, and Populations County Coalition/Agency Name Populationa Prescription Drugs Berkeley Kennedy Center 210,898 Darlington Rubicon 67,234 Dorchester Dorchester Alcohol and Drug Commission 153,773 Greenville Phoenix Center 498,766 Horry Shoreline Behavioral Health Services 322,342 TOTAL   1,253,013 Impaired Driving Barnwell Axis 1 Center 21,483 Chester Hazel Pittman 32,181 Jasper New Life Center 28,465 Marlboro Trinity Behavioral Care 26,945 Orangeburg Tri-County Commission on Alcohol and Drug Abuse 87,903 196,977 a US Census Bureau, American Fact Finder, https://factfinder.census.gov, estimates for July 1, 2016. 86% 14%

Focus on ECHO: Limitations of Site-Specific Data Power in numbers…it is harder to observe and attribute change when looking at data from any individual site, especially for relatively low-incident events. No identified comparison data. Some of the data are proxies of the construct—may not be the most ideal indicator (e.g., social access).

Limitations of Site Specific Data At a site-specific level, you are monitoring outcomes, not attributing changes in outcomes to your interventions. Even in the aggregate, we will have challenges attributing change to the collective interventions.

Outcome Evaluation/Monitoring Definition Outcome/effectiveness evaluation measures program effects in the target population by assessing the program outcomes or outcome objectives that the program is to achieve (CDC) Outcome monitoring is the routine measurement and reporting of indicators of the results of a program’s efforts in the social domain it is accountable for approving…A prerequisite for outcome monitoring is the identification of the outcomes that can reasonably be expected to produce. (Rossi, Freeman, & Lipsey)

Impact Evaluation Time-Based Definition Impact evaluation assesses program effectiveness in achieving its ultimate goals (generally beyond the study population). (CDC) Impact evaluation has the same goal as outcome evaluation, although it is conducted later than outcome evaluation—typically years after the end of a program—to determine the lasting effect on the target population. (K4Health) Impact: The positive and negative, primary and secondary long- term effects produced by a development intervention, directly or indirectly, intended or unintended. (INTRAC)

Impact Evaluation Attribution-Based Definition Impact is the degree to which the outcomes observed are attributable to the program activities. (Stanford Social Innovation Review) Impact evaluation, also referred to as outcome evaluation…(Rossi et al.) Key Point: Outcome and impact language can be confusing and imprecise. That’s why it’s important to develop a clear logic model.

Logic Model Establishes connections between outcomes and your strategies Tells the story of how your strategies may be influencing outcomes

Statewide Logic Model

Statewide Logic Model Short term outcome

Statewide Logic Model

Statewide Logic Model Long term outcome

Statewide Logic Model

Statewide Logic Model Impacts

Statewide Logic Model

Rx: Retail Access Count of drugs (pills) dispensed Count of drugs (prescriptions) dispensed Count of recipients of Rx drugs

Rx Retail Access: Quantity of Pills Dispensed

Rx Social Access Number of prescriptions received from others Amount of unused prescriptions in circulation Use of safe disposal/safe storage

Rx Social Access: Number of People that Received Deterra

Rx Perceptions of Risk Risk of harm from using Rx drugs not prescribed to them and improper use of drugs prescribed to them Data Sources: YRBS (annual, but statewide) CTC (biennial) PIRE community survey (pre-post)

Use of Rx drugs without a Rx Improper use of prescribed drugs Rx Consumption Use of Rx drugs without a Rx Improper use of prescribed drugs Data Sources: YRBS (annual, but statewide) CTC (biennial) PIRE community survey (pre-post)

Rx Consequences Drug-related overdose deaths Drug-related hospital ED visits

Rx Consequences: Drug Overdose Deaths

Rx Drug Story Community is implementing: Community is experiencing: Prescriber education (provide numbers) Consumer education about, and tools for, safe storage and disposal (provide numbers) Media campaigns about risks associated with nonmedical use of Rx drugs (provide #s) Community is experiencing: Reductions in number of pills prescribed (with exceptions) Increases in number of people with safe disposal kits Reductions in prescription drug-related deaths (increases in heroin and fentanyl) Cannot make causal attributions but can link strategies with outcomes you are monitoring. Story should be accurately reflected and should provide insights into your strategies.

Statewide Logic Model

Impaired Driving: Perceptions of Risk Risk of being injured in an crash Risk of injuring someone in a crash Risk of arrest/citation Data Sources: YRBS (annual, but statewide) CTC (biennial) PIRE community survey (pre-post) Arrest/citation rates

Impaired Driving Perceptions: Arrests/Citations per 100 Cars

Impaired Driving Consumption Drinking and driving Driving with someone who drank Data Sources: YRBS (annual, but statewide) CTC (biennial) PIRE community survey (pre-post)

Impaired Driving Consequences DUI crashes (by age group) Percent crashes that are DUI (by age) DUI fatalities Rate of DUI fatalities Single vehicle nighttime crashes

Impaired Driving Consequences: DUI Crashes Under 21

Impaired Driving Consequences: DUI Crashes Total

Impaired Driving Story Community is implementing: Saturation patrols (provide numbers) Safety checkpoints (provide numbers)Prescriber education Routine enforcement (provide numbers) Media campaigns about risks associated with impaired driving (provide numbers) Community is experiencing: Increases in the rates of citations/arrests Decreases in DUI crashes Cannot make causal attributions but can link strategies with outcomes you are monitoring. Story should be accurately reflected and should provide insights into your strategies.