1. Quarterly Meeting June 4, 2019 Al Stein-Seroussi
State of South Carolina Department of Alcohol and Other Drug Abuse Services Partnerships for Success - ECHO Project Outcome and Impact Evaluation
Quarterly Meeting
June 4, 2019
Al Stein-Seroussi

2. To review concepts related to outcome and impact evaluations
Purpose of the Session
To review concepts related to outcome and impact evaluations
To review MIS spreadsheets in related to outcome and impact evaluations
(Not discussing outputs)

3. SC ECHO Sites

4. ECHO Coalitions, Counties, and Populations
County
Coalition/Agency Name
Populationa
Prescription Drugs
Berkeley
Kennedy Center
210,898
Darlington
Rubicon
67,234
Dorchester
Dorchester Alcohol and Drug Commission
153,773
Greenville
Phoenix Center
498,766
Horry
Shoreline Behavioral Health Services
322,342
TOTAL
1,253,013
Impaired Driving
Barnwell
Axis 1 Center
21,483
Chester
Hazel Pittman
32,181
Jasper
New Life Center
28,465
Marlboro
Trinity Behavioral Care
26,945
Orangeburg
Tri-County Commission on Alcohol and Drug Abuse
87,903
196,977
a US Census Bureau, American Fact Finder, estimates for July 1, 2016.
86%
14%

5. Focus on ECHO: Limitations of Site-Specific Data
Power in numbers…it is harder to observe and attribute change when looking at data from any individual site, especially for relatively low-incident events.
No identified comparison data.
Some of the data are proxies of the construct—may not be the most ideal indicator (e.g., social access).

6. Limitations of Site Specific Data
At a site-specific level, you are monitoring outcomes, not attributing changes in outcomes to your interventions. Even in the aggregate, we will have challenges attributing change to the collective interventions.

7. Outcome Evaluation/Monitoring Definition
Outcome/effectiveness evaluation measures program effects in the target population by assessing the program outcomes or outcome objectives that the program is to achieve (CDC)
Outcome monitoring is the routine measurement and reporting of indicators of the results of a program’s efforts in the social domain it is accountable for approving…A prerequisite for outcome monitoring is the identification of the outcomes that can reasonably be expected to produce. (Rossi, Freeman, & Lipsey)

8. Impact Evaluation Time-Based Definition
Impact evaluation assesses program effectiveness in achieving its ultimate goals (generally beyond the study population). (CDC)
Impact evaluation has the same goal as outcome evaluation, although it is conducted later than outcome evaluation—typically years after the end of a program—to determine the lasting effect on the target population. (K4Health)
Impact: The positive and negative, primary and secondary long- term effects produced by a development intervention, directly or indirectly, intended or unintended. (INTRAC)

9. Impact Evaluation Attribution-Based Definition
Impact is the degree to which the outcomes observed are attributable to the program activities. (Stanford Social Innovation Review)
Impact evaluation, also referred to as outcome evaluation…(Rossi et al.)
Key Point: Outcome and impact language can be confusing and imprecise. That’s why it’s important to develop a clear logic model.

10. Logic Model
Establishes connections between outcomes and your strategies
Tells the story of how your strategies may be influencing outcomes

11. Statewide Logic Model

12. Statewide Logic Model
Short term outcome

13. Statewide Logic Model

14. Statewide Logic Model
Long term outcome

15. Statewide Logic Model

16. Statewide Logic Model
Impacts

17. Statewide Logic Model

18. Rx: Retail Access
Count of drugs (pills) dispensed
Count of drugs (prescriptions) dispensed
Count of recipients of Rx drugs

19. Rx Retail Access: Quantity of Pills Dispensed

20. Rx Social Access
Number of prescriptions received from others
Amount of unused prescriptions in circulation
Use of safe disposal/safe storage

21. Rx Social Access: Number of People that Received Deterra

22. Rx Perceptions of Risk
Risk of harm from using Rx drugs not prescribed to them and improper use of drugs prescribed to them
Data Sources:
YRBS (annual, but statewide)
CTC (biennial)
PIRE community survey (pre-post)

23. Use of Rx drugs without a Rx Improper use of prescribed drugs
Rx Consumption
Use of Rx drugs without a Rx
Improper use of prescribed drugs
Data Sources:
YRBS (annual, but statewide)
CTC (biennial)
PIRE community survey (pre-post)

24. Rx Consequences
Drug-related overdose deaths
Drug-related hospital ED visits

25. Rx Consequences: Drug Overdose Deaths

26. Rx Drug Story Community is implementing: Community is experiencing:
Prescriber education (provide numbers)
Consumer education about, and tools for, safe storage and disposal (provide numbers)
Media campaigns about risks associated with nonmedical use of Rx drugs (provide #s)
Community is experiencing:
Reductions in number of pills prescribed (with exceptions)
Increases in number of people with safe disposal kits
Reductions in prescription drug-related deaths (increases in heroin and fentanyl)
Cannot make causal attributions but can link strategies with outcomes you are monitoring.
Story should be accurately reflected and should provide insights into your strategies.

27. Statewide Logic Model

28. Impaired Driving: Perceptions of Risk
Risk of being injured in an crash
Risk of injuring someone in a crash
Risk of arrest/citation
Data Sources:
YRBS (annual, but statewide)
CTC (biennial)
PIRE community survey (pre-post)
Arrest/citation rates

29. Impaired Driving Perceptions: Arrests/Citations per 100 Cars

30. Impaired Driving Consumption
Drinking and driving
Driving with someone who drank
Data Sources:
YRBS (annual, but statewide)
CTC (biennial)
PIRE community survey (pre-post)

31. Impaired Driving Consequences
DUI crashes (by age group)
Percent crashes that are DUI (by age)
DUI fatalities
Rate of DUI fatalities
Single vehicle nighttime crashes

32. Impaired Driving Consequences: DUI Crashes Under 21

33. Impaired Driving Consequences: DUI Crashes Total

34. Impaired Driving Story
Community is implementing:
Saturation patrols (provide numbers)
Safety checkpoints (provide numbers)Prescriber education
Routine enforcement (provide numbers)
Media campaigns about risks associated with impaired driving (provide numbers)
Community is experiencing:
Increases in the rates of citations/arrests
Decreases in DUI crashes
Cannot make causal attributions but can link strategies with outcomes you are monitoring.
Story should be accurately reflected and should provide insights into your strategies.