Building Evaluation Capacity for Prescription Opioid Overdose Prevention: Lessons From a Multi-state Initiative Session Chair: Sarah Bacon, PhD Presenters: Kari Cruz, MPH, Bradley Biggers, MPH, Emilia Pasalic, MPH, and Aleta Christensen, MPH Presentation to the American Evaluation Association Annual Conference October 29, 2016

500,000 4x as many opioid deaths drug overdose deaths since 2000 in 2014 as 1999

For every Rx opioid overdose death in 2011, there were... SAMHSA NSDUH, DAWN, TEDS data sets.

Overdose Prevention In States (OPIS) Three competitive cooperative agreements, all to state health departments or their bona fide agents Prevention for States Data Driven Prevention Initiative Enhanced Surveillance for Opioid Morbidity and Mortality

1 2 4 3 Prevention for States Program Enhance and Maximize PDMPs Move toward universal PDMP registration and use Make PDMPs easier to use and access Move toward a real-time PDMP Expand and improve proactive reporting Conduct public health surveillance with PDMP Enhance and Maximize PDMPs Community or Health System Interventions State Policy Evaluation Rapid Response Projects Implement or improve opioid prescribing interventions for insurers, health systems, or pharmacy benefit managers. This includes: Prior authorization, prescribing rules, academic detailing, CCPs, PRRs, Enhance adoption of opioid prescribing guidelines 1 2 Prevention for States Program 4 3 Allow states to move on quick, flexible projects to respond to changing circumstances on the ground and move fast to capitalize on new prevention opportunities. Build evidence base for policy prevention strategies that work like pain clinic laws and regulations, or naloxone access laws

Evaluation Capacity Building: Tools for State Partners National Center for Injury Prevention and Control Division of Unintentional Injury Prevention

Evaluation Officers

Current Funded States IL TX NY IA CA SD ND NE MN MI ID MS KS IN LA WY NM AL WI NC OK GA VA CO KY HI MT UT SC OH OR TN PA WV WA AZ ME FL MO NV AK NH VT MA RI CT NJ DE MD DC AR Funded State: Prevention for States (PfS) Funded State: Data-Driven Prevention Initiative (DDPI)

CDC State Support Teams Program Consultant Evaluation Officer Science Officer

State Prescription Drug Overdose Teams Program Manager Evaluator Epidemiologist

Technical Assistance Tailored assistance that is purposeful, responsive, science-based, and systematic in building their infrastructure and capacity around implementation and evaluation to improve outcomes associated with opioid misuse, abuse and overdose prevention programming. 

Evaluation Tools Monitoring and Evaluation Tool (MET) Evaluation plans Indicators

Capacity Building: Monitoring and Evaluation Tool (MET) National Center for Injury Prevention and Control Division of Analysis, Research, and Practice Integration

Monitoring and Evaluation Tool (MET) Background Microsoft Access Database Started in 2011 Developed primarily by Deloitte Supports Prevention for States (PfS) and three other CDC cooperative agreements

Improve responsiveness of CDC staff in support of PfS awardees Identify capacity of state prescription drug overdose programs Identify emerging needs and gaps so CDC can deliver timely technical assistance

Capacity Building: Developing Evaluation Plans with State Partners National Center for Injury Prevention and Control Division of Unintentional Injury Prevention

Evaluation Plan Development Process Template developed States develop evaluation plans Evaluation Plans submitted by state to CDC Review criteria Review worksheets Gems! Evaluation Plan reviewed by CDC Evaluation Officer Feedback form Formal discussion TA COP CDC provides feedback to state National Center for Injury Prevention and Control Division of Unintentional Injury Prevention

Evaluation Plan Template Template sections: Project specific logic model Detailed evaluation plan for project years one and two Narrative description of evaluation plan for years three and four

Evaluation Plan Template Development National Center for Injury Prevention and Control Division of Unintentional Injury Prevention

Evaluation Plan Review Processes Common criteria worksheets Logic model review worksheet Evaluation Plan review worksheet Evaluation Plan Narrative review worksheet Gathering resources: Evaluation Gems! Evaluation Plan Feedback Template National Center for Injury Prevention and Control Division of Unintentional Injury Prevention

Logic Model Review Worksheet National Center for Injury Prevention and Control Division of Unintentional Injury Prevention

Evaluation Plan review worksheet

Evaluation Plan Narrative Review Worksheet

Evaluation Gems! An internal tracking spreadsheet prompts Evaluation Officers to document: Details of exemplary evaluation plans Resources received from the state Additional Comments

Evaluation Plan Feedback Template

Evaluation Profiles

Lessons Learned Both reviewers and reviewees appreciate clear criteria! Limit the amount of mandatory forms Systematic approach = Proactive technical assistance

Capacity Building: Indicator Development National Center for Injury Prevention and Control Division of Unintentional Injury Prevention

Evaluation Capacity and Data Use Data Sources: Vital statistics death data Emergency department data Hospital discharge data Prescription drug monitoring program (PDMP) data

Developing the Indicators – Challenges Definition How to calculate (numerator & denominator) Limitations

Prevention for States Indicator Support Toolkit Contents: Background, Introduction, and Purpose Indicator Considerations Preparing the datasets Indicator quick guide One-pagers for each indicator Sample SAS code

Developing the Indicators – Engaging Stakeholders

Indicator and Toolkit Pilot 2 piloting phases Funded state partners ASTHO Pilot though email and webinars

Progress and Next Steps

1600 Clifton Road NE, Atlanta, GA 30333 For more information please contact Centers for Disease Control and Prevention 1600 Clifton Road NE, Atlanta, GA 30333 Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348 E-mail: cdcinfo@cdc.gov Web: www.cdc.gov The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.